{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolution Nets for MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References:\n",
    "\n",
    "Some of the images and the content I used came from this great couple of blog posts \\[1\\] [https://adeshpande3.github.io/adeshpande3.github.io/]() and \\[2\\] the  terrific book, [\"Neural Networks and Deep Learning\"](http://neuralnetworksanddeeplearning.com/) by Michael Nielsen. (**Strongly recommend**)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A convolutional neural network (CNN, or ConvNet) is a type of **feed-forward** artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"imgs/convnets_cover.png\" width=\"70%\" />\n",
    "\n",
    "> source: https://flickrcode.files.wordpress.com/2014/10/conv-net2.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Structure of a CNN\n",
    "\n",
    "> A more detailed overview of what CNNs do would be that you take the image, pass it through a series of convolutional, nonlinear, pooling (downsampling), and fully connected layers, and get an output. As we said earlier, the output can be a single class or a probability of classes that best describes the image. \n",
    "\n",
    "source: [1]\n",
    "\n",
    "## Convolutional Layer\n",
    "\n",
    "The first layer in a CNN is always a **Convolutional Layer**.\n",
    "\n",
    "<img src=\"http://deeplearning.net/software/theano/_images/same_padding_no_strides.gif\" width=\"50%\">\n",
    "\n",
    "**Reference**: [http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html](http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Typical CNN Structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A traditional **convolutional neural network** architecture, there are other layers that are interspersed between these conv layers.\n",
    "\n",
    "<img src=\"https://adeshpande3.github.io/assets/Table.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " After some ReLU layers, it is customary to apply a **pooling layer** (aka *downsampling layer*).\n",
    " \n",
    " Example of a MaxPooling filter\n",
    " \n",
    " <img src=\"imgs/MaxPool.png\" width=\"80%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other options for pooling layers are average pooling and L2-norm pooling. \n",
    "\n",
    "This kind of layer serves two main purposes: reduce the amount of parameters; controlling overfitting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# CNN in Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Keras** has an extensive support for Convolutional Layers:\n",
    "\n",
    "- 1D Convolutional Layers;\n",
    "- 2D Convolutional Layers;\n",
    "- 3D Convolutional Layers;\n",
    "- Depthwise Convolution;\n",
    "- Transpose Convolution;\n",
    "- ....\n",
    "\n",
    "The corresponding `keras` package is `keras.layers.convolutional`.\n",
    "\n",
    "Take a look at the [Convolutional Layers](https://keras.io/layers/convolutional/) documentation to know more about Conv Layers that are missing in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convolution1D\n",
    "\n",
    "```python\n",
    "from keras.layers.convolutional import Conv1D\n",
    "\n",
    "Conv1D(filters, kernel_size, strides=1, padding='valid', \n",
    "       dilation_rate=1, activation=None, use_bias=True, \n",
    "       kernel_initializer='glorot_uniform', bias_initializer='zeros', \n",
    "       kernel_regularizer=None, bias_regularizer=None, \n",
    "       activity_regularizer=None, kernel_constraint=None, \n",
    "       bias_constraint=None)\n",
    "```\n",
    "\n",
    "#### Arguments:\n",
    "\n",
    "<ul>\n",
    "<li><strong>filters</strong>: Integer, the dimensionality of the output space\n",
    "    (i.e. the number output of filters in the convolution).</li>\n",
    "<li><strong>kernel_size</strong>: An integer or tuple/list of a single integer,\n",
    "    specifying the length of the 1D convolution window.</li>\n",
    "<li><strong>strides</strong>: An integer or tuple/list of a single integer,\n",
    "    specifying the stride length of the convolution.\n",
    "    Specifying any stride value != 1 is incompatible with specifying\n",
    "    any <code>dilation_rate</code> value != 1.</li>\n",
    "<li><strong>padding</strong>: One of <code>\"valid\"</code>, <code>\"causal\"</code> or <code>\"same\"</code> (case-insensitive).\n",
    "    <code>\"causal\"</code> results in causal (dilated) convolutions, e.g. output[t]\n",
    "    does not depend on input[t+1:]. Useful when modeling temporal data\n",
    "    where the model should not violate the temporal order.\n",
    "    See <a href=\"https://arxiv.org/abs/1609.03499\">WaveNet: A Generative Model for Raw Audio, section 2.1</a>.</li>\n",
    "<li><strong>dilation_rate</strong>: an integer or tuple/list of a single integer, specifying\n",
    "    the dilation rate to use for dilated convolution.\n",
    "    Currently, specifying any <code>dilation_rate</code> value != 1 is\n",
    "    incompatible with specifying any <code>strides</code> value != 1.</li>\n",
    "<li><strong>activation</strong>: Activation function to use\n",
    "    (see <a href=\"https://keras.io/activations/\">activations</a>).\n",
    "    If you don't specify anything, no activation is applied\n",
    "    (ie. \"linear\" activation: <code>a(x) = x</code>).</li>\n",
    "<li><strong>use_bias</strong>: Boolean, whether the layer uses a bias vector.</li>\n",
    "<li><strong>kernel_initializer</strong>: Initializer for the <code>kernel</code> weights matrix\n",
    "    (see <a href=\"https://keras.io/initializers/\">initializers</a>).</li>\n",
    "<li><strong>bias_initializer</strong>: Initializer for the bias vector\n",
    "    (see <a href=\"https://keras.io/initializers/\">initializers</a>).</li>\n",
    "<li><strong>kernel_regularizer</strong>: Regularizer function applied to\n",
    "    the <code>kernel</code> weights matrix\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>bias_regularizer</strong>: Regularizer function applied to the bias vector\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>activity_regularizer</strong>: Regularizer function applied to\n",
    "    the output of the layer (its \"activation\").\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>kernel_constraint</strong>: Constraint function applied to the kernel matrix\n",
    "    (see <a href=\"https://keras.io/constraints/\">constraints</a>).</li>\n",
    "<li><strong>bias_constraint</strong>: Constraint function applied to the bias vector\n",
    "    (see <a href=\"https://keras.io/constraints/\">constraints</a>).</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Convolution operator for filtering neighborhoods of **one-dimensional inputs**. When using this layer as the first layer in a model, either provide the keyword argument `input_dim` (int, e.g. 128 for sequences of 128-dimensional vectors), or `input_shape` (tuple of integers, e.g. (10, 128) for sequences of 10 vectors of 128-dimensional vectors)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example\n",
    "\n",
    "```python\n",
    "\n",
    "# apply a convolution 1d of length 3 to a sequence with 10 timesteps,\n",
    "# with 64 output filters\n",
    "model = Sequential()\n",
    "model.add(Conv1D(64, 3, padding='same', input_shape=(10, 32)))\n",
    "# now model.output_shape == (None, 10, 64)\n",
    "\n",
    "# add a new conv1d on top\n",
    "model.add(Conv1D(32, 3, padding='same'))\n",
    "# now model.output_shape == (None, 10, 32)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convolution2D\n",
    "\n",
    "```python\n",
    "from keras.layers.convolutional import Conv2D\n",
    "\n",
    "Conv2D(filters, kernel_size, strides=(1, 1), padding='valid', \n",
    "       data_format=None, dilation_rate=(1, 1), activation=None, \n",
    "       use_bias=True, kernel_initializer='glorot_uniform', \n",
    "       bias_initializer='zeros', kernel_regularizer=None, \n",
    "       bias_regularizer=None, activity_regularizer=None, \n",
    "       kernel_constraint=None, bias_constraint=None)\n",
    "```\n",
    "\n",
    "#### Arguments:\n",
    "\n",
    "<ul>\n",
    "<li><strong>filters</strong>: Integer, the dimensionality of the output space\n",
    "    (i.e. the number output of filters in the convolution).</li>\n",
    "<li><strong>kernel_size</strong>: An integer or tuple/list of 2 integers, specifying the\n",
    "    width and height of the 2D convolution window.\n",
    "    Can be a single integer to specify the same value for\n",
    "    all spatial dimensions.</li>\n",
    "<li><strong>strides</strong>: An integer or tuple/list of 2 integers,\n",
    "    specifying the strides of the convolution along the width and height.\n",
    "    Can be a single integer to specify the same value for\n",
    "    all spatial dimensions.\n",
    "    Specifying any stride value != 1 is incompatible with specifying\n",
    "    any <code>dilation_rate</code> value != 1.</li>\n",
    "<li><strong>padding</strong>: one of <code>\"valid\"</code> or <code>\"same\"</code> (case-insensitive).</li>\n",
    "<li><strong>data_format</strong>: A string,\n",
    "    one of <code>channels_last</code> (default) or <code>channels_first</code>.\n",
    "    The ordering of the dimensions in the inputs.\n",
    "    <code>channels_last</code> corresponds to inputs with shape\n",
    "    <code>(batch, height, width, channels)</code> while <code>channels_first</code>\n",
    "    corresponds to inputs with shape\n",
    "    <code>(batch, channels, height, width)</code>.\n",
    "    It defaults to the <code>image_data_format</code> value found in your\n",
    "    Keras config file at <code>~/.keras/keras.json</code>.\n",
    "    If you never set it, then it will be \"channels_last\".</li>\n",
    "<li><strong>dilation_rate</strong>: an integer or tuple/list of 2 integers, specifying\n",
    "    the dilation rate to use for dilated convolution.\n",
    "    Can be a single integer to specify the same value for\n",
    "    all spatial dimensions.\n",
    "    Currently, specifying any <code>dilation_rate</code> value != 1 is\n",
    "    incompatible with specifying any stride value != 1.</li>\n",
    "<li><strong>activation</strong>: Activation function to use\n",
    "    (see <a href=\"https://keras.io/activations/\">activations</a>).\n",
    "    If you don't specify anything, no activation is applied\n",
    "    (ie. \"linear\" activation: <code>a(x) = x</code>).</li>\n",
    "<li><strong>use_bias</strong>: Boolean, whether the layer uses a bias vector.</li>\n",
    "<li><strong>kernel_initializer</strong>: Initializer for the <code>kernel</code> weights matrix\n",
    "    (see <a href=\"https://keras.io/initializers/\">initializers</a>).</li>\n",
    "<li><strong>bias_initializer</strong>: Initializer for the bias vector\n",
    "    (see <a href=\"https://keras.io/initializers/\">initializers</a>).</li>\n",
    "<li><strong>kernel_regularizer</strong>: Regularizer function applied to\n",
    "    the <code>kernel</code> weights matrix\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>bias_regularizer</strong>: Regularizer function applied to the bias vector\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>activity_regularizer</strong>: Regularizer function applied to\n",
    "    the output of the layer (its \"activation\").\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>kernel_constraint</strong>: Constraint function applied to the kernel matrix\n",
    "    (see <a href=\"https://keras.io/constraints/\">constraints</a>).</li>\n",
    "<li><strong>bias_constraint</strong>: Constraint function applied to the bias vector\n",
    "    (see <a href=\"https://keras.io/constraints/\">constraints</a>).</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example\n",
    "Assuming \n",
    "``keras.backend.image_data_format == \"channels_last\"``\n",
    "```python\n",
    "\n",
    "# apply a 3x3 convolution with 64 output filters on a 256x256 image:\n",
    "model = Sequential()\n",
    "model.add(Conv2D(64, (3, 3), padding='same', \n",
    "          input_shape=(3, 256, 256)))\n",
    "# now model.output_shape == (None, 256, 256, 64)\n",
    "\n",
    "# add a 3x3 convolution on top, with 32 output filters:\n",
    "model.add(Conv2D(32, (3, 3), padding='same'))\n",
    "# now model.output_shape == (None, 256, 256, 32)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensions of Conv Filters in Keras\n",
    "\n",
    "The complex structure of ConvNets *may* lead to a representation that is challenging to understand.\n",
    "\n",
    "Of course, the dimensions vary according to the dimension of the Convolutional filters (e.g. 1D, 2D)\n",
    "\n",
    "### Convolution1D\n",
    "\n",
    "**Input Shape**:\n",
    "\n",
    "**3D** tensor with shape: (`batch_size`, `steps`, `input_dim`).\n",
    "\n",
    "**Output Shape**:\n",
    "\n",
    "**3D** tensor with shape: (`batch_size`, `new_steps`, `filters`).\n",
    "\n",
    "### Convolution2D\n",
    "\n",
    "**Input Shape**:\n",
    "\n",
    "**4D** tensor with shape: \n",
    "\n",
    "- (`batch_size`, `channels`, `rows`, `cols`) if `image_data_format='channels_first'`\n",
    "- (`batch_size`, `rows`, `cols`, `channels`) if `image_data_format='channels_last'`\n",
    "\n",
    "**Output Shape**:\n",
    "\n",
    "**4D** tensor with shape:\n",
    "\n",
    "- (`batch_size`, `filters`, `new_rows`, `new_cols`) \n",
    "if `image_data_format='channels_first'`\n",
    "- (`batch_size`, `new_rows`, `new_cols`, `filters`) if `image_data_format='channels_last'`\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Deep Learning models can take quite a bit of time to run, particularly if GPU isn't used. \n",
    "\n",
    "In the interest of time, you could sample a subset of observations (e.g. $1000$) that are a particular number of your choice (e.g. $6$) and $1000$ observations that aren't that particular number (i.e. $\\neq 6$). \n",
    "\n",
    "We will build a model using that and see how it performs on the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "#Import the required libraries\n",
    "import numpy as np\n",
    "np.random.seed(1338)\n",
    "\n",
    "from keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation, Flatten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.layers.convolutional import Conv2D\n",
    "from keras.layers.pooling import MaxPooling2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.utils import np_utils\n",
    "from keras.optimizers import SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#Load the training and testing data\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "X_test_orig = X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Very Important: \n",
    "When dealing with images & convolutions, it is paramount to handle `image_data_format` properly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    shape_ord = (1, img_rows, img_cols)\n",
    "else:  # channel_last\n",
    "    shape_ord = (img_rows, img_cols, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess and Normalise Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "X_train = X_train.reshape((X_train.shape[0],) + shape_ord)\n",
    "X_test = X_test.reshape((X_test.shape[0],) + shape_ord)\n",
    "\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "\n",
    "X_train /= 255\n",
    "X_test /= 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(1338)  # for reproducibilty!!\n",
    "\n",
    "# Test data\n",
    "X_test = X_test.copy()\n",
    "Y = y_test.copy()\n",
    "\n",
    "# Converting the output to binary classification(Six=1,Not Six=0)\n",
    "Y_test = Y == 6\n",
    "Y_test = Y_test.astype(int)\n",
    "\n",
    "# Selecting the 5918 examples where the output is 6\n",
    "X_six = X_train[y_train == 6].copy()\n",
    "Y_six = y_train[y_train == 6].copy()\n",
    "\n",
    "# Selecting the examples where the output is not 6\n",
    "X_not_six = X_train[y_train != 6].copy()\n",
    "Y_not_six = y_train[y_train != 6].copy()\n",
    "\n",
    "# Selecting 6000 random examples from the data that \n",
    "# only contains the data where the output is not 6\n",
    "random_rows = np.random.randint(0,X_six.shape[0],6000)\n",
    "X_not_six = X_not_six[random_rows]\n",
    "Y_not_six = Y_not_six[random_rows]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Appending the data with output as 6 and data with output as <> 6\n",
    "X_train = np.append(X_six,X_not_six)\n",
    "\n",
    "# Reshaping the appended data to appropraite form\n",
    "X_train = X_train.reshape((X_six.shape[0] + X_not_six.shape[0],) + shape_ord)\n",
    "\n",
    "# Appending the labels and converting the labels to \n",
    "# binary classification(Six=1,Not Six=0)\n",
    "Y_labels = np.append(Y_six,Y_not_six)\n",
    "Y_train = Y_labels == 6 \n",
    "Y_train = Y_train.astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11918, 28, 28, 1) (11918,) (10000, 28, 28, 1) (10000,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape, Y_labels.shape, X_test.shape, Y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Converting the classes to its binary categorical form\n",
    "nb_classes = 2\n",
    "Y_train = np_utils.to_categorical(Y_train, nb_classes)\n",
    "Y_test = np_utils.to_categorical(Y_test, nb_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A simple CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# -- Initializing the values for the convolution neural network\n",
    "\n",
    "nb_epoch = 2  # kept very low! Please increase if you have GPU\n",
    "\n",
    "batch_size = 64\n",
    "# number of convolutional filters to use\n",
    "nb_filters = 32\n",
    "# size of pooling area for max pooling\n",
    "nb_pool = 2\n",
    "# convolution kernel size\n",
    "nb_conv = 3\n",
    "\n",
    "# Vanilla SGD\n",
    "sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 1: Model Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv), padding='valid', \n",
    "                 input_shape=shape_ord))  # note: the very first layer **must** always specify the input_shape\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 2: Compile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=sgd,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Step 3: Fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 8s - loss: 0.2321 - acc: 0.9491 - val_loss: 0.1276 - val_acc: 0.9616\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.1065 - acc: 0.9666 - val_loss: 0.0933 - val_acc: 0.9685\n"
     ]
    }
   ],
   "source": [
    "hist = model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "                 epochs=nb_epoch, verbose=1, \n",
    "                 validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fdb2c0235f8>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VGX2wPHvSSOkQkioARIpCaGH0KSEANJEEGUVbCsW\nhLUi7IrurrK6rv52AQFFERXUtSALFkCaSpcakB5CQg819BIgBN7fHzPCJAQYwkwmM3M+zzMPmTvv\nnZwLOmfufd9zrhhjUEoppW7Ex9UBKKWUcg+aMJRSStlFE4ZSSim7aMJQSillF00YSiml7KIJQyml\nlF00YSillLKLJgyllFJ20YShlFLKLn6uDsCRIiMjTUxMjKvDUEopt7F69erDxpgoe8Z6VMKIiYkh\nNTXV1WEopZTbEJFd9o7VS1JKKaXsoglDKaWUXTRhKKWUsotHzWEopTzDhQsXyMrK4ty5c64OxWME\nBgYSHR2Nv79/kd9DE4ZSqsTJysoiNDSUmJgYRMTV4bg9YwxHjhwhKyuL2NjYIr+PXpJSSpU4586d\no1y5cposHEREKFeu3C2fsWnCUEqVSJosHMsRf5+aMIB3f8lgze5jrg5DKaVKNK9PGCdyLvDFil3c\n8/5SBk9ex6FTOsmmlLc7cuQIjRo1olGjRlSsWJEqVapcfp6bm2vXe/Tr14/09PTrjhk7dixffvml\nI0IuFmKMcXUMDpOUlGSKUul9+nwe783L5JMl2ynl58vzHWrxx9tjCPDz+nyqlEukpaVRp04dV4cB\nwLBhwwgJCWHIkCH5thtjMMbg4+M+nxOF/b2KyGpjTJI9+7vPkTpRSCk/hnaNZ+6gZJrGlOXNmWl0\nHb2IRVuzXR2aUqoEyczMJCEhgQcffJC6deuyf/9++vfvT1JSEnXr1uX111+/PLZ169asXbuWvLw8\nypQpw9ChQ2nYsCEtW7bk0KFDAPztb39j1KhRl8cPHTqUZs2aERcXx9KlSwE4c+YM9957LwkJCfTu\n3ZukpCTWrl1b/AePLqvNJzYymIn9mvFL2kFen7GZRyaspFNCBf7ePYGqEUGuDk8pr/SP6ZvYvO+k\nQ98zoXIYr91Vt0j7btmyhc8//5ykJMuX8rfffpuIiAjy8vJISUmhd+/eJCQk5NvnxIkTJCcn8/bb\nb/Piiy8yYcIEhg4detV7G2NYuXIl06ZN4/XXX2f27Nm8++67VKxYkalTp7Ju3ToSExOLFLcj6BlG\nITrUqcDcQW35c+c4FmccpsPIhYycm87Z3IuuDk0p5WI1atS4nCwAvv76axITE0lMTCQtLY3Nmzdf\ntU/p0qXp2rUrAE2aNGHnzp2Fvvc999xz1ZglS5bQp08fABo2bEjdukVLdI6gZxjXUMrPl6dTanJP\nYhXemrmFMfMymbI6i7/emUC3+hV1yZ9SxaSoZwLOEhwcfPnnjIwMRo8ezcqVKylTpgwPPfRQobUO\nAQEBl3/29fUlLy+v0PcuVarUDce4kp5h3ECl8NKM6duYyU+1JDwogKe/WsMDH60g/cApV4emlHKx\nkydPEhoaSlhYGPv372fOnDkO/x2tWrVi8uTJAGzYsKHQM5jiomcYdmoWG8GMZ1vz1crdjJibTrcx\ni3m4RXUGdaxNeFDRe7MopdxXYmIiCQkJxMfHU716dVq1auXw3/Hss8/yyCOPkJCQcPkRHh7u8N9j\nD11WWwTHzuQy4qd0vlqxmzJBAfylcxx/SKqKr49eplLKEUrSslpXy8vLIy8vj8DAQDIyMujUqRMZ\nGRn4+d389/1bXVarZxhFUDY4gH/eXZ++zaoxbNomhn67gS9X7GZYj7o0qV7W1eEppTzI6dOn6dCh\nA3l5eRhj+PDDD4uULBxBE8YtqFs5nMlPtWTaun38a2Ya936wlHsSqzC0azzlQwNdHZ5SygOUKVOG\n1atXuzoMQCe9b5mI0LNRFeYNbsfAdjWYsW4/7YcvZPyibeTmXXJ1eEop5TCaMBwkuJQfL3WJZ86g\ntjSLjeBfM7fQZfQiFmq1uFLKQ2jCcLDYyGAmPNqUCY8mcemS4Y8TVvLk56nsPpLj6tCUUuqWODVh\niEgXEUkXkUwRuaoOXkQeFJH1IrJBRJaKSEPr9qoiMl9ENovIJhF53plxOkP7+ArMGdSWl7rE82vm\nYTq+s5ARc9PJyS15xThKKWUPpyUMEfEFxgJdgQSgr4gkFBi2A0g2xtQH3gDGW7fnAYONMQlAC+Dp\nQvYt8Ur5+TKwXQ3mDW5Ht3oVeXdeJh1HLGTG+n140nJmpTxNSkrKVUV4o0aNYuDAgdfcJyQkBIB9\n+/bRu3fvQse0a9eOGy39HzVqFDk5V65IdOvWjePHj9sbulM58wyjGZBpjNlujMkFJgE9bQcYY5Ya\nY36/c9FyINq6fb8xZo3151NAGlDFibE6VcXwQEb1acz/BrSkTFAAz3z1G30/Ws6WA45tqKaUcoy+\nffsyadKkfNsmTZpE3759b7hv5cqVmTJlSpF/d8GEMXPmTMqUKVPk93MkZyaMKsAem+dZXP9D/3Fg\nVsGNIhIDNAZWFLaTiPQXkVQRSc3OLtkTzE1jIpj+bGv+eXc9thw4RbfRi3nth42cyLng6tCUUjZ6\n9+7Njz/+ePlmSTt37mTfvn00btyYDh06kJiYSP369fnhhx+u2nfnzp3Uq1cPgLNnz9KnTx/q1KlD\nr169OHv27OVxAwcOvNwW/bXXXgNgzJgx7Nu3j5SUFFJSUgCIiYnh8OHDAIwcOZJ69epRr169y23R\nd+7cSZ06dXjyySepW7cunTp1yvd7HKlE1GGISAqWhNG6wPYQYCrwgjGm0K/jxpjxWC9lJSUllfjr\nPL4+wkMtqtO9QSVGzN3Kf5fvYtq6ffy5czz3N9VqcaWuMmsoHNjg2PesWB+6vn3NlyMiImjWrBmz\nZs2iZ8+eTJo0ifvuu4/SpUvz3XffERYWxuHDh2nRogU9evS4ZjPSDz74gKCgINLS0li/fn2+1uRv\nvvkmERERXLx4kQ4dOrB+/Xqee+45Ro4cyfz584mMjMz3XqtXr2bixImsWLECYwzNmzcnOTmZsmXL\nkpGRwddff81HH33Efffdx9SpU3nooYcc83dlw5lnGHuBqjbPo63b8hGRBsDHQE9jzBGb7f5YksWX\nxphvnRinS5QJCuCNu+sx49k21CofyivfbaDn2CWs3nXU1aEppch/Wer3y1HGGF555RUaNGhAx44d\n2bt3LwcPHrzmeyxatOjyB3eDBg1o0KDB5dcmT55MYmIijRs3ZtOmTTdsKrhkyRJ69epFcHAwISEh\n3HPPPSxevBiA2NhYGjVqBFy/ffqtcuYZxiqglojEYkkUfYAHbAeISDXgW+BhY8xWm+0CfAKkGWNG\nOjFGl0uoHMY3T7Vg+vr9/OvHNO79YBn3NLZWi4dptbhS1zsTcKaePXsyaNAg1qxZQ05ODk2aNOHT\nTz8lOzub1atX4+/vT0xMTKHtzG9kx44dDB8+nFWrVlG2bFkeffTRIr3P735viw6W1ujOuiTltDMM\nY0we8AwwB8uk9WRjzCYRGSAiA6zDXgXKAe+LyFoR+X35QCvgYaC9dftaEenmrFhdTUTo0bAyvwxO\n5umUGsxYv5+U4Qv4cKFWiyvlKiEhIaSkpPDYY49dnuw+ceIE5cuXx9/fn/nz57Nr167rvkfbtm35\n6quvANi4cSPr168HLG3Rg4ODCQ8P5+DBg8yadWX6NjQ0lFOnrr59Qps2bfj+++/JycnhzJkzfPfd\nd7Rp08ZRh2sXp85hGGNmAjMLbBtn8/MTwBOF7LcE8LqL+cGl/Phz53j+0KQq//xxM2/N2sI3q/bw\n6l0JtIsr7+rwlPI6ffv2pVevXpcvTT344IPcdddd1K9fn6SkJOLj46+7/8CBA+nXrx916tShTp06\nNGnSBLDcOa9x48bEx8dTtWrVfG3R+/fvT5cuXahcuTLz58+/vD0xMZFHH32UZs2aAfDEE0/QuHFj\np11+Koy2Ny/B5qcf4vXpm9lx+Awd65Tn790TqF4u+MY7KuXmtL25c9xqe3NtDVKCpcSVZ84LbRna\nNZ5l245wx8hF/GfOFq0WV0q5hCaMEi7Az4cByTWYN6QddzaoxNj52+gwYiHT1mm1uFKqeGnCcBMV\nwgJ55/5GTBnQkojgAJ77+jfuH7+ctP1aLa48k34hcixH/H1qwnAzSTERTHumNf/qVZ+Mg6e4c8xi\nXv1hI8dzcl0dmlIOExgYyJEjRzRpOIgxhiNHjhAYeGtL9XXS240dz8ll5E9b+WL5LsJL+zOkcxx9\nmlbTanHl9i5cuEBWVtYt1Sao/AIDA4mOjsbf3z/f9puZ9NaE4QHS9p9k2LRNrNhxlLqVw/hHj7ok\nxUS4OiyllBvQVVJepk6lMCb1b8G7fRtz9Ewuvcct44VJv3HwpH47U0o5jiYMDyEi3GWtFn8mpSYz\nNxwgZfgCPliwjfN5F10dnlLKA2jC8DBBAX4M6RzHTy+25fYakfzf7C10GbWY+emHXB2aUsrNacLw\nUNXLBfPxH5P4tF9TBOg3cRWPf7qKnYfPuDo0pZSb0oTh4drFlWf2C215uWs8y7cfodM7i/j37C2c\nOa/V4kqpm6MJwwsE+PnwVHIN5g9pR/cGlXh/gaVa/Ie1e3Wdu1LKbpowvEj5sEBG3t+IqQNbEhka\nwPOT1nL/h8vZvE+rxZVSN6YJwws1qR7BD0+35q176pOZfZru7y7mb99v4NgZrRZXSl2bJgwv5esj\n9G1WjfmD2/FIyxi+WrGblBEL+GL5Li5e0stUSqmracLwcuFB/gzrUZeZz7chvmIof/t+I3e9u4RV\nO/Xe4kqp/DRhKADiK4bx9ZMteO+BxhzLyeUP45bx/KTfOHBCq8WVUhaaMNRlIkL3BpZq8Wfb12TW\nxgO0H7GA9xdkarW4Usq5CUNEuohIuohkisjQQl5/UETWi8gGEVkqIg3t3Vc5T1CAH4M7xfHzoGRa\n1Yzk37PT6fzOIuZtOejq0JRSLuS0hCEivsBYoCuQAPQVkYQCw3YAycaY+sAbwPib2Fc5WbVyQXz0\nSBKfPdYMHx/hsU9TeezTVezQanGlvJIzzzCaAZnGmO3GmFxgEtDTdoAxZqkx5pj16XIg2t59VfFJ\nrh3F7Ofb8tdudVi54yid3lnI27O0Wlwpb+PMhFEF2GPzPMu67VoeB2YVcV/lZAF+PjzZ9jbmDU6m\nR8MqjFu4jfYjFmi1uFJepERMeotICpaE8VIR9u0vIqkikpqdne344FQ+5cMCGXFfQ6YOvJ3yoYE8\nP2kt9324jE37Trg6NKWUkzkzYewFqto8j7Zuy0dEGgAfAz2NMUduZl8AY8x4Y0ySMSYpKirKIYGr\nG2tSvSw/PN2Kt++pz7bsM9z17hL++p1WiyvlyZyZMFYBtUQkVkQCgD7ANNsBIlIN+BZ42Biz9Wb2\nVa7n4yP0sakWn7RqD+2GL+C/y3ZqtbhSHshpCcMYkwc8A8wB0oDJxphNIjJARAZYh70KlAPeF5G1\nIpJ6vX2dFau6NZerxZ9rQ0KlMP7+wya6v7uEFduP3HhnpZTbEE+asExKSjKpqamuDsOrGWOYtfEA\nb/6Yxt7jZ+nRsDIvd4unUnhpV4emlCqEiKw2xiTZM7ZETHorzyEidKtfiZ9fTOa5DrWYvekA7Ycv\nZOx8rRZXyt1pwlBOUTrAlxfvqM0vLybTplYk/5mTTqd3FvFLmlaLK+WuNGEop6oaEcT4R5L4/LFm\n+PkIj3+WSr+JK9mefdrVoSmlbpImDFUs2taOYvYLbfnbnXVYtfMYnUct4q1ZaZzWanGl3IYmDFVs\n/H19eKLNbcwbkkzPRlX4cOF22g9fwHe/ZWm1uFJuQBOGKnblQwMZ/oeGfPen26kUHsigb9bRe9wy\nNu7VanGlSjJNGMplGlcry3d/asW/723AzsNnuOu9Jbzy3QaOarW4UiWSJgzlUj4+wn1NqzJvSDv6\n3R7LN6v2kDJ8AZ8v20nexUuuDk8pZUMThioRwkv78+pdCcx6vg11K4fxqrVafLlWiytVYmjCUCVK\n7QqhfPlEcz54MJFT5/LoM345z379G/uOn3V1aEp5PU0YqsQREbpaq8Wf71CLuZsO0GHEQt6bl8G5\nC1otrpSraMJQJVbpAF8G3VGbn19MJrl2FMPnbqXTO4v4afNBXYarlAtowlAlXtWIIMY93IQvHm9u\nufPf56k8OnEV27RaXKlipQlDuY3WtSKZ9Xwb/t49gTW7jtFl1CLempnGqXMXXB2aUl5BE4ZyK/6+\nPjzeOpZ5Q9pxd6MqfLhoO+1HLOTbNVlc0ps2KeVUmjCUW4oKLcV//tCQ759uReXwQF6cvI7e45Zq\ntbhSTqQJQ7m1RlXLWKrFezdg99Ec7npvCS9/u4Ejp8+7OjSlPI4mDOX2fHyE+5Is1eKPtYplcqql\nWvzTX3dotbhSDqQJQ3mMsEB//t49gdnPt6F+dDjDpm+m+7tLWLZNq8WVcgRNGMrj1KoQyhePN2fc\nQ5Zq8b4fLefpr9ZotbhSt8ipCUNEuohIuohkisjQQl6PF5FlInJeRIYUeG2QiGwSkY0i8rWIBDoz\nVuVZRIQu9Srxy+BkBnWszc+bD9J+xALe/UWrxZUqKqclDBHxBcYCXYEEoK+IJBQYdhR4DhheYN8q\n1u1Jxph6gC/Qx1mxKs8V6O/L8x1r8cvgZFLiyjPip63c8c5C5m46oNXiSt0kZ55hNAMyjTHbjTG5\nwCSgp+0AY8whY8wqoLDKKz+gtIj4AUHAPifGqjxcdNkgPnioCV8+0ZxAP1/6/3c1f5y4isxDWi2u\nlL2cmTCqAHtsnmdZt92QMWYvlrOO3cB+4IQxZm5hY0Wkv4ikikhqdnb2LYasPF2rmpHMfL4Nr3ZP\n4Lfdlmrxf2m1uFJ2KZGT3iJSFsvZSCxQGQgWkYcKG2uMGW+MSTLGJEVFRRVnmMpN+fv68FjrWOYP\nace9idF8tNhSLT51tVaLK3U9zkwYe4GqNs+jrdvs0RHYYYzJNsZcAL4FbndwfMrLRYaU4v96N+D7\nP7WiSpnSDP6fpVp8fdZxV4emVInkzISxCqglIrEiEoBl0nqanfvuBlqISJCICNABSHNSnMrLNaxa\nhm8H3s5/rNXiPcf+ytCp67VaXKkC/Jz1xsaYPBF5BpiDZZXTBGPMJhEZYH19nIhUBFKBMOCSiLwA\nJBhjVojIFGANkAf8Box3VqxK+fgIf0iqSud6FRnzcwafLt3Jjxv28+IdtXm4RXX8fEvk1VulipV4\n0tLCpKQkk5qa6uowlAfIPHSKf0zfzOKMw9SuEMKwHnW5vUakq8NSyuFEZLUxJsmesfq1SalC1Cwf\nyuePNePDh5uQk3uRBz5awdNfrmGvVosrL6YJQ6lrEBE6163Izy8m8+Idtflly0E6jFjAGK0WV15K\nE4ZSNxDo78tzHWrx84vJdIivwMifttJx5ELmaLW48jKaMJSyU3TZIMY+mMhXTzQnKMCXp/67mkcm\nrCTz0ClXh6ZUsdCEodRNur1mJDOfa8NrdyWwds9xuoxazD9nbNZqceXxNGEoVQR+vj70axXLgiHt\n6N0kmk9+3UHK8IX8L3WPVosrj6UJQ6lbUC6kFG/f24Afnm5F1YjS/HnKeu75YCnr9mi1uPI8mjCU\ncoAG0WWYOuB2hv+hIVnHznL3+7/y0pT1HNZqceVBNGEo5SA+PkLvJtHMH5LMk21uY+qaLFKGL2DC\nkh1c0HuLKw+gCUMpBwsN9OeVbnWY/UJbGlUtw+szNnPnmMUszTzs6tCUuiV2JQwRqSEipaw/txOR\n50SkjHNDU8q91SwfwuePNWP8w004e+EiD3y8goFfrCbrWI6rQ1OqSOw9w5gKXBSRmliaAFYFvnJa\nVEp5CBGhU92K/DQomcF31GZ++iE6jFjIqJ+3arW4cjv2JoxLxpg8oBfwrjHmz0Al54WllGcJ9Pfl\n2Q61+GVwOzomVGDUzxl0GLGQ2Rv3a7W4chv2JowLItIX+CMww7rN3zkhKeW5qpQpzdgHEvn6yRaE\nlPJjwBdrePiTlWQc1GpxVfLZmzD6AS2BN40xO0QkFviv88JSyrO1rFGOH59rzbC7ElifdZyuoxfz\nxozNnNRqcVWC3fT9MKz3265qjFnvnJCKTu+HodzRkdPnGT53K5NW7aZccAB/6RJP78RofHzE1aEp\nL+Dw+2GIyAIRCRORCCx3wftIREbeSpBKKYtyIaV46576THu6NdUigvjLlPX0+mApa7VaXJUw9l6S\nCjfGnATuAT43xjQHOjovLKW8T/3ocKYMuJ2R9zVk3/Gz3D32V/4yZR3Zp7RaXJUM9iYMPxGpBNzH\nlUlvpZSD+fgI9yRGM39IO55qexvf/baX9sMX8PHi7VotrlzO3oTxOjAH2GaMWSUitwEZN9pJRLqI\nSLqIZIrI0EJejxeRZSJyXkSGFHitjIhMEZEtIpImIi3tjFUptxdSyo+XrdXijauX5Z8/ptF19GKW\nZGi1uHKdm570tvuNRXyBrcAdQBawCuhrjNlsM6Y8UB24GzhmjBlu89pnwGJjzMciEgAEGWOue1FX\nJ72VJzLG8HPaId6YsZndR3PoUrcif72zDlUjglwdmvIAzpj0jhaR70TkkPUxVUSib7BbMyDTGLPd\nGJMLTAJ62g4wxhwyxqwC8q0lFJFwoC3wiXVc7o2ShVKeSkS4I6ECcwe1ZUin2izcmk3HkQt55yet\nFlfFy95LUhOBaUBl62O6ddv1VAH22DzPsm6zRyyQDUwUkd9E5GMRCbZzX6U8UqC/L8+0r8Uvg5O5\nI6ECo3+xVIvP2qDV4qp42JswoowxE40xedbHp0CUE+PyAxKBD4wxjYEzwFVzIAAi0l9EUkUkNTs7\n24khKVUyVC5TmvceSGRS/xaEBvox8Ms1PPTJCrZqtbhyMnsTxhEReUhEfK2Ph4AjN9hnL5Ymhb+L\ntm6zRxaQZYxZYX0+BUsCuYoxZrwxJskYkxQV5cwcplTJ0uK2csx4tjWv96zLxr0n6Tp6Ma9P38yJ\ns1otrpzD3oTxGJYltQeA/UBv4NEb7LMKqCUisdZJ6z5YLmvdkDHmALBHROKsmzoAm6+zi1Jeyc/X\nh0daxjB/SDvub1qViUt30H74Ar5ZtVvvLa4crsirpETkBWPMqBuM6QaMAnyBCcaYN0VkAIAxZpyI\nVARSgTDgEnAaSDDGnBSRRsDHQACwHehnjDl2vd+nq6SUt9u49wSvTdvE6l3HaBgdzrAedWlcrayr\nw1Il2M2skrqVhLHbGFOtSDs7iSYMpSzLcL9fu5e3Zm7h0Knz9G4SzUtd4okKLeXq0FQJ5PBltdf6\nPbewr1LKSUSEXo2jmTekHU8l38YPa7VaXDnGrSQMvUCqVAkWUsqPl7vWYc4LbWkSc6VafHGGriZU\nRXPdhCEip0TkZCGPU1jqMZRSJdxtUSFMfLQpn/wxiQsXL/HwJyt56r+p7Dmq9xZXN8fvei8aY0KL\nKxCllPOICB3qVKBVzUg+WbKD9+Zl0jF9IU8l12Bgcg1KB/i6OkTlBm7lkpRSys0E+vvydEpN5g1J\npnPdioz5JYOOIxcyU6vFlR00YSjlhSqFl2ZM38Z8Y60W/9OXa3jw4xWkH9BqcXVtmjCU8mLNrdXi\nb/Ssy6Z9J+k2ZjHDpm3SanFVKE0YSnk5P18fHm4Zw4Ih7ejTtCqfLdtJyvAFTFqp1eIqP00YSikA\nygYH8Gav+kx/pjU1ooIZ+u0G7n7/V9bsvm6DBeVFNGEopfKpVyWcyU+1ZHSfRhw8eY573l/K4Mnr\nOHTqnKtDUy6mCUMpdRURoWejKvwyuB0D29Vg2rq9tB++kI8WbSc3T6vFvZUmDKXUNYWU8uOlLvHM\nHZRM05iyvDkzjS6jF7Foq1aLeyNNGEqpG4qNDGZiv2ZMeDSJS5cMj0xYyZOfp7L7iFaLexNNGEop\nu7WPr8CcQW35S5c4fs08TMd3FjJibjpnc/Xe4t5AE4ZS6qaU8vPlT+1qMm9wO7rWq8i78zLpMGIB\nM9bv02pxD6cJQylVJBXDAxndpzGTn2pJeFAAz3z1G30/Ws6WAyddHZpyEk0YSqlb0iw2wlItfnc9\nthw4xZ1jlliqxXO0WtzTaMJQSt0yXx/h4RbVmT+4HQ80q8bny3aSMmIBX6/czUWtFvcYmjCUUg5T\nNjiAN+6ux/RnW1MzKoSXv93A3WN/ZfUurRb3BJowlFIOV7dyON881YLRfRqRfeo8936wlBe/Wcuh\nk1ot7s6cmjBEpIuIpItIpogMLeT1eBFZJiLnRWRIIa/7ishvIjLDmXEqpRzvSrV4Mn9qV4MZ6/eT\nMnwBHy7cptXibsppCUNEfIGxQFcgAegrIgkFhh0FngOGX+NtngfSnBWjUsr5gkv58Zcu8cwd1JYW\nt5XjrVlb6DJ6EQu1WtztOPMMoxmQaYzZbozJBSYBPW0HGGMOGWNWAVctpxCRaOBO4GMnxqiUKiYx\nkcF88mhTJj7aFGPgjxNW8sRnWi3uTpyZMKoAe2yeZ1m32WsU8BfguueuItJfRFJFJDU7W7+xKFXS\npcSXZ/YLbXipSzxLt1mqxYfPSScnN8/VoakbKJGT3iLSHThkjFl9o7HGmPHGmCRjTFJUVFQxRKeU\nulWl/HwZ2K4G84e04876lXhvfiYdRixk+jqtFi/JnJkw9gJVbZ5HW7fZoxXQQ0R2YrmU1V5EvnBs\neEopV6sQFsg79zdiyoCWlA0K4Nmvf6PP+OWk7ddq8ZLImQljFVBLRGJFJADoA0yzZ0djzMvGmGhj\nTIx1v3nGmIecF6pSypWSYiKY/mxr3uxVj60HT3HnmMW89sNGjufkujo0ZcPPWW9sjMkTkWeAOYAv\nMMEYs0lEBlhfHyciFYFUIAy4JCIvAAnGGP16oZSX8fURHmxenTvrV2LkT1v57/JdTFu3jyGd4+jT\ntBq+PuLqEL2eeNL1wqSkJJOamurqMJRSDrB530mGTd/Eyh1HqVcljH/0qEuT6hGuDsvjiMhqY0yS\nPWNL5KS3UkolVA7jm/4tGNO3MYdP5XLvB8sY9M1aDmq1uMtowlBKlVgiQo+GlZk3JJmnU2rw4/r9\ntB++gHFs8WAyAAASgElEQVRaLe4SmjCUUiVeUIAff+4cz08vtqVljXK8PWsLXUYtYn76IVeH5lU0\nYSil3Eb1csF8/MemTOzXFIB+E1fx+Ker2Hn4jIsj8w6aMJRSbiclrjyzX2jLy13jWb79CJ3eWcR/\n5mzhzHmtFncmTRhKKbcU4OfDU8k1mDekHd0bVGLs/G10GLGQH9bu1WpxJ9GEoZRyaxXCAhlprRYv\nFxLA85PWcv/45Wzep+VcjqYJQynlEZJiIpj2TGv+1as+GQdP0f3dxfz9e60WdyRNGEopj+HrIzzQ\nvBoLhqTwcIvqfLliF+2GL+CL5bv03uIOoAlDKeVxwoP8+UfPevz4XBviKoTyt+83cte7S1i186ir\nQ3NrmjCUUh6rTqUwJvVvwXsPNOZYTi5/GLeMFyb9xoETWi1eFE5rPuhWpjwOQREQWRui4iAyDkLK\ng2izM6XcnYjQvUFl2seX54MF2/hw0Xbmbj7Is+1r8VjrGEr5+bo6RLehzQcv5sGEzpCdDrmnrmwP\nDLckjqja1j/jLAmlTHXw0RMzpdzV7iM5vPHjZn7afJDYyGBe7Z5ASnx5V4flMjfTfFATxu+MgVP7\nLYkjOx0Op0P2VsufZ2xu/epXGiJr5k8iUXEQUQP8AhxzIEopp1uQfojXZ2xme/YZ2seX59XuCcRE\nBrs6rGKnCcPRco7C4a3WRLL1SkI5vvvKGPGFiNgCZyW1LQmlVKjjY1JK3bLcvEt8unQHo3/O4MJF\nw+NtYnkmpSbBpbznar0mjOKSewYOZ+RPItlb4eg2uGTToiAsukASsZ6dBEcWX6xKqWs6dPIcb8/e\nwrdr9lIhrBSvdKtDj4aVES+Yx9SE4WoXL8DRHdYEkp7/7ORCzpVxpSPyX9b6PaGERes8iVIusHrX\nMYZN28SGvSdoGlOWYT3qUrdyuKvDcipNGCXVpUtwMuvK3IhtMjlrsz7cPxgia12dTCJiwdffdfEr\n5QUuXjL8L3UP/56TzvGcXB5oXo3Bd8RRNtgz5yg1YbijM4evnmzP3mpJML/z8YeI265euRVZGwKC\nXBe7Uh7oRM4F3vnZcm/x0EA/BneK44Fmnndv8RKTMESkCzAa8AU+Nsa8XeD1eGAikAj81Rgz3Lq9\nKvA5UAEwwHhjzOgb/T63ThjXcv6U9Sxka/5kcnQHmItXxpWpdvXKrcjalvoSpVSRbTlwkmHTNrF8\n+1HqVLLcW7xZrOf8f1UiEoaI+AJbgTuALGAV0NcYs9lmTHmgOnA3cMwmYVQCKhlj1ohIKLAauNt2\n38J4ZMK4lrzzcHT71Su3DmdAnk0Va3BUISu34iCsshYmKmUnYwwzNxzgzR83s+/EOXo0rMwr3epQ\nMTzQ1aHdsptJGM5cO9YMyDTGbLcGNQnoCVz+0DfGHAIOicidtjsaY/YD+60/nxKRNKCK7b5ez68U\nlK9jedi6dNGy3Lfgyq2NU+HciSvjAkILX7lVpjr4es+SQqXsISLc2aCStVo8k3GLtvNz2kGeTqnJ\nE21ivaZa3JmfDFWAPTbPs4DmN/smIhIDNAZWOCQqT+djrQeJiIXana9sNwZOH7p6sn3bPFj31ZVx\nvgFQrmb+y1pRcVCuFvi7/7cppW5F6QBfXuwUR+8mVfnnj5v5z5x0Jqfu4dXuCXSoU8HV4Tldif4q\nKSIhwFTgBWNMoXdDEZH+QH+AatWqFWN0bkYEQitYHrFt87927sTVK7f2r4O0aWAu/f4GULY6RMVf\nvQw40LOXHSpVULVyQYx/JIlFW7MZNn0Tj3+WSkpcFH/vnsBtUSGuDs9pnDmH0RIYZozpbH3+MoAx\n5q1Cxg4DTv8+h2Hd5g/MAOYYY0ba8zu9ag6jOFw4B0cyC6zcSrdsu2hzU5qQilev3IqKg5AKOk+i\nPF5u3iU+W7qT0b9kcD7vIo+3vo1n2tckxE2qxUvKpLcflknvDsBeLJPeDxhjNhUydhg2CUMs5ZWf\nAUeNMS/Y+zs1YRSTi3lwfFfhy4DtauBYzXLpTCkPcujUOf49O50pq7OoEFaKl7vWoWejkl8tXiIS\nhjWQbsAoLMtqJxhj3hSRAQDGmHEiUhFIBcKAS8BpIAFoACwGNli3A7xijJl5vd+nCcPFbBs4Ht4K\n2Vuu0cAx0DInUnDSvVwNy2S+Um5szW5Ltfj6rBMkVbdUi9erUnIv25aYhFHcNGGUYNrAUXmRS5cM\n/1u9h3/PTudoTi59m1VjSKc4IkpgtbgmDOU+cnPgSEb+OZLs9EIaOFa5euVWpLWBYwk/5Vfe68TZ\nC4z6eSufL9tFSCk/BneqzQPNquHnW3J6xWnCUO7vmg0cM+DCmSvjSpctfOWWNnBUJUj6gVMMm7aJ\nZduPEF8xlH/0qEvz28q5OixAE4arw1DOdOkSnNx7ZZI9e8s1GjgGWRo4Fpx0j7hNGzgqlzDGMGvj\nAd78MY29x89yV8PKvNItnkrhpV0alyYM5Z3sauDoZ7k74lUNHGtBgPfdbU0Vv7O5F/lg4TbGLdyG\nrwjPtK/J461jCfR3zcpBTRhK2Tp/yuZGV1uu3cAxvFrh7VK0gaNygj1Hc3jzxzRmbzpA9XJB/P3O\nBDrUKV/sy3A1YShlj7xcy+T6jRo4BkVa5km0gaNygsUZ2fxj+mYyD50muXYUr91VvNXimjCUuhWX\nLsGJ3TZzJDaXuAo2cCzsRldlY7SBo7opFy5aq8V/zuBc3kUeax3Ls+1rFUu1uCYMpZzhWg0cD2+1\nFCz+zjfAOk9SoFVKuZrg79oJTlWyZZ86z79nb+F/q7OICi3Fy13j6dW4ilMvU2nCUKq4nTthuZSV\nvSV/Mjm+6+oGjoW1SyldxqXhq5LlN2u1+LqsEzSpXpZ/OLFaXBOGUiVFoQ0ct1qKFfM1cKxgU0ei\nDRyVpVp8yuos/m/2Fo7m5NKnaTX+3Nnx1eKaMJQq6S5dhGM7r165VbCBY6nwQlZu1bbc6EobOHqF\nE2cvMPrnDD5btpPgAF8Gd4rjweaOqxbXhKGUu7qqgaPNn2cOXRmnDRy9TsbBUwybvolfMy3V4q/d\nVZeWNW69WlwThlKeyN4GjmVjrl65FaUNHD2BMYY5mw7wxgxLtfidDSrx1251qFym6IspNGEo5U0K\na+B4eCsc2QaXLlwZF1r56pVb2sDRLZ3NvciHi7bxwYJt+IjwdEoN+retQYDfzV+m0oShlLI0cDy2\n8+qVW4U1cCxs5VZ4VW3gWMLtOZrDv2amsS37ND8+1wb/IsxraMJQSl1bwQaOtn/mHLkyThs4uo1T\n5y4QGli0f5ObSRhajqqUt/HxgTJVLY+aHfO/VlgDx11LYcNkm/39LEmj4BxJZG1t4OgiRU0WN0sT\nhlLqiuBIyyOmVf7t509bLmkVnHRPn1WggWPVAklEGzh6Ek0YSqkbKxUCVRItD1vXauC481fIO3tl\nXFBkgcl2659hVXTC3Y1owlBKFZ1fAJSvY3nYsm3gaLtya9N3cO74lXEBIdYGjvHawNENOPVfRES6\nAKMBX+BjY8zbBV6PByYCicBfjTHD7d1XKVWC+fhYPvTLxkDtTle2X6uB4/YFsO7rK+MuN3As5EZX\n2sDRZZyWMETEFxgL3AFkAatEZJoxZrPNsKPAc8DdRdhXKeVuRCC0guUR2zb/a5cbONpMuu9fD2nT\n8zdwLFOtwP1JtIFjcXHmGUYzINMYsx1ARCYBPYHLH/rGmEPAIRG582b3VUp5mMBwiE6yPGxdq4Hj\n9gVw8fyVcSEVClm5FQehFXWexEGcmTCqAHtsnmcBzR29r4j0B/oDVKtW7eajVEqVbP6BULGe5WEr\nXwNHm8tb677RBo5O4vazSsaY8cB4sBTuuTgcpVRx8fG1NFssVwPiul7Zfq0GjhlzYe0XV8b5BVpu\nalVw5Va5mtrA8RqcmTD2AlVtnkdbtzl7X6WUNxOx3G89rDLUSMn/2tljV6/c2ptqWb2F9fumWCfs\nf58f+f0SV2QtCAwr7qMpUZyZMFYBtUQkFsuHfR/ggWLYVymlCle6LFRrbnnYulYDx8yfC2ngWPDy\nVhwER3nFPInTEoYxJk9EngHmYFkaO8EYs0lEBlhfHyciFYFUIAy4JCIvAAnGmJOF7eusWJVSXi4g\nCCo1tDxsXW7gmH4lkWSnw29f5G/gGFim8JVbHtbAUZsPKqXUzbqZBo7lal69civiNkvRYwmgzQeV\nUsqZrtvA8UiBwsQtsGsZbPifzf5+UDY2/xyJGzRw1IShlFKOFFwOgm+H6rfn334zDRwLrtyKjLO8\nr4tpwlBKqeJw3QaO2/PPkfzeVj5fA8dyBVZuFX8DR00YSinlSn4BUD7e8rB1Mw0cK9aHfrOcnjg0\nYSilVEl0vQaOZ7Lzr9zKO1csZxmaMJRSyp2IQEh5yyO2TbH+as9ZIKyUUsqpNGEopZSyiyYMpZRS\ndtGEoZRSyi6aMJRSStlFE4ZSSim7aMJQSillF00YSiml7OJR7c1FJBvYVcTdI4HDDgzHHegxez5v\nO17QY75Z1Y0xUfYM9KiEcStEJNXenvCeQo/Z83nb8YIeszPpJSmllFJ20YShlFLKLpowrhjv6gBc\nQI/Z83nb8YIes9PoHIZSSim76BmGUkopu3hVwhCRLiKSLiKZIjK0kNdFRMZYX18vIomFvY87seOY\nH7Qe6wYRWSoiDV0RpyPd6JhtxjUVkTwR6V2c8TmDPccsIu1EZK2IbBKRhcUdo6PZ8d92uIhMF5F1\n1mPu54o4HUVEJojIIRHZeI3Xnf/5ZYzxigfgC2wDbgMCgHVAQoEx3YBZgAAtgBWujrsYjvl2oKz1\n567ecMw24+YBM4Hero67GP6dywCbgWrW5+VdHXcxHPMrwP9Zf44CjgIBro79Fo65LZAIbLzG607/\n/PKmM4xmQKYxZrsxJheYBPQsMKYn8LmxWA6UEZFKxR2oA93wmI0xS40xx6xPlwPRxRyjo9nz7wzw\nLDAVOFScwTmJPcf8APCtMWY3gDHG3Y/bnmM2QKiICBCCJWHkFW+YjmOMWYTlGK7F6Z9f3pQwqgB7\nbJ5nWbfd7Bh3crPH8ziWbyju7IbHLCJVgF7AB8UYlzPZ8+9cGygrIgtEZLWIPFJs0TmHPcf8HlAH\n2AdsAJ43xlwqnvBcwumfX3pPbwWAiKRgSRitXR1LMRgFvGSMuWT58ukV/IAmQAegNLBMRJYbY7a6\nNiyn6gysBdoDNYCfRGSxMeaka8NyX96UMPYCVW2eR1u33ewYd2LX8YhIA+BjoKsx5kgxxeYs9hxz\nEjDJmiwigW4ikmeM+b54QnQ4e445CzhijDkDnBGRRUBDwF0Thj3H3A9421gu8GeKyA4gHlhZPCEW\nO6d/fnnTJalVQC0RiRWRAKAPMK3AmGnAI9bVBi2AE8aY/cUdqAPd8JhFpBrwLfCwh3zbvOExG2Ni\njTExxpgYYArwJzdOFmDff9s/AK1FxE9EgoDmQFoxx+lI9hzzbixnVIhIBSAO2F6sURYvp39+ec0Z\nhjEmT0SeAeZgWWExwRizSUQGWF8fh2XFTDcgE8jB8g3Fbdl5zK8C5YD3rd+484wbN26z85g9ij3H\nbIxJE5HZwHrgEvCxMabQ5ZnuwM5/5zeAT0VkA5aVQy8ZY9y2i62IfA20AyJFJAt4DfCH4vv80kpv\npZRSdvGmS1JKKaVugSYMpZRSdtGEoZRSyi6aMJRSStlFE4ZSSim7aMJQ6gZE5KK1y+vvj2t2wC3C\ne8dcq/uoUiWN19RhKHULzhpjGrk6CKVcTc8wlCoiEdkpIv+23ktkpYjUtG6PEZF51nsS/GKtpkdE\nKojId9b7M6wTkdutb+UrIh9Z79kwV0RKW8c/JyKbre8zyUWHqdRlmjCUurHSBS5J3W/z2gljTH0s\nnVFHWbe9C3xmjGkAfAmMsW4fAyw0xjTEcl+DTdbttYCxxpi6wHHgXuv2oUBj6/sMcNbBKWUvrfRW\n6gZE5LQxJqSQ7TuB9saY7SLiDxwwxpQTkcNAJWPMBev2/caYSBHJBqKNMedt3iMG+MkYU8v6/CXA\n3xjzT2srj9PA98D3xpjTTj5Upa5LzzCUujXmGj/fjPM2P1/kytzincBYLGcjq0RE5xyVS2nCUOrW\n3G/z5zLrz0uxdE8FeBBYbP35F2AggIj4ikj4td5URHyAqsaY+cBLQDiWu8Yp5TL6jUWpGystImtt\nns82xvy+tLasiKzHcpbQ17rtWWCiiPwZyOZK19DngfEi8jiWM4mBwLXaT/sCX1iTigBjjDHHHXZE\nShWBzmEoVUTWOYwkd26ZrdTN0EtSSiml7KJnGEoppeyiZxhKKaXsoglDKaWUXTRhKKWUsosmDKWU\nUnbRhKGUUsoumjCUUkrZ5f8B7mYrlyR2uvYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdb309ee5f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FeX1+PHPIexr2LcQEpAlYQ/XgBuKuOBKoVRBLQIq\nxSr6a2sr8rVfKHah1lpxqYqKta2K1n6xwQXqggWrLImGJSFICAhhDWuAELKd3x8zCTcRkkvI5OYm\n5/168cqdeWZ5HtB7MmdmniOqijHGGFPV6gW7A8YYY2onCzDGGGM8YQHGGGOMJyzAGGOM8YQFGGOM\nMZ6wAGOMMcYTFmCMMcZ4wgKMMcYYT1iAMcYY44n6we5AMLVr106joqKC3Q1jjAkpSUlJB1S1fUXb\n1ekAExUVRWJiYrC7YYwxIUVEvg1kO0uRGWOM8YQFGGOMMZ6wAGOMMcYTFmCMMcZ4wgKMMcYYT1iA\nMcYY4wkLMMYYYzxRp9+DMcaYOqUwH/auhx2roWVn6DfW09NZgDHGmNrq5BHIXAs7VsHO1bArCfJz\nnLb+4y3AGGOMCYAqHPnWCSbFAWX/JkBBwqDTAIibBN2GQeRwaNnF8y5ZgDHGmFDkn+7aucr5eXyv\n09aoJURc6FyhdBsGXYdCo+bV3kULMMYYEwrKS3eFR0L0CIgcBt2GQ4cYqBcW3P5iAcYYY2oeVTi8\n3QkkNSTdVRkWYIwxJtgK82HPejfV5QaU4/ucthqS7qoMCzDGGFPdTh6GnWtP3zvZlQQFJ5228EiI\nvrzGpbsqw9MAIyKjgflAGPCyqs4r094aWAj0BHKBqaq60W0LB14G+gPqtn0pIm8BfdxDhANHVHWw\niEQBm4DNbtsqVZ3u4fCMMaZipdJdXzoBJSuNknRX54EwdPLpgNKyc5A7XHU8CzAiEgY8B1wNZAJr\nRSRBVVP9NpsFJKvqWBHp624/ym2bDyxV1fEi0hBoCqCqt/qd44/AUb/jbVXVwV6NyRhjKlRuuqsV\ndLsQ+n/fCShdh0LDZsHtr4e8vIKJB9JVNQNARBYBYwD/ABMLzANQ1TQRiRKRjjhXMyOAyW5bHpDn\nf3AREeAW4EoPx2CMMeWrKN3V44rTN+Pb9w3ZdFdleBlgugI7/ZYzgWFltlkHjANWikg80B2IAAqB\nLOBVERkEJAEPquoJv30vA/ap6ha/ddEikoxzVfOoqq4s2ykRmQZMA4iMjDyP4Rlj6hxVOLyt9Lsn\nWZuctlqe7qqMYN/knwfMd4PCBuBrnOBSH4gDZqjqahGZD8wEfum370TgTb/lPUCkqh4UkaHAuyLS\nT1Wz/U+oqguABQA+n089GpcxpjYoyHNfZlx1OqCc2O+01bF0V2V4GWB2Ad38liPcdSXcL/8pUJLy\n2gZk4NxvyVTV1e6m7+AEGNxt6+Nc+Qz1O9Yp4JT7OUlEtgK9gcQqHZUxpvY6eRh2rvF7mfErv3RX\nd+g50i/dFQP1bEL68ngZYNYCvUQkGiewTABu89/AfVIsx73Hcjewwg062SKyU0T6qOpmnBv//vdu\nrgLSVDXT71jtgUOqWigiPYBeOMHKGGO+q7x0V7360Gkg+KacDigtOgW3vyHIswCjqgUicj+wDOcx\n5YWqmiIi0932F4AY4DURUSAFuMvvEDOA190nyDJwr3RcEyidHgPnoYC5IpIPFAHTVfWQB0MzxoSi\nCtNd8TDg+869k65DoWHT4Pa3FhDVunsbwufzaWKiZdCMqZW+k+5KgoJcp611lBNIim/Gt+9r6a5z\nICJJquqraLtg3+Q3xpjzpwqHMkrP3ZWV5rSVpLumOqmubsMs3VVNLMAYY0JPQR7sWef3MuOa0+mu\nxq0gIh4GjLd0V5BZgDHG1Hw5h9yp6t2pVnZ/VTrd1fNKS3edg+0HTlBQpFzQwdtJMy3AGGNqlorS\nXZ0Hge8uN6BYuutcnDhVwLPL03ll5TYu6tmW16bGe3o+CzDGmOD6TrprNZzIctoat3KCyIAfOPdP\nusRZuqsSVJV/Je/mdx9uYl/2KcbFdWXm6L6en9cCjDGmeuUccu6ZFD8qXCrdFQ0XXHX63ZN2fSzd\ndZ427jrK7IQUkr49zICurfjz7UMZ2r11tZzbAowxxjvF6S7/d08OuBU1vpPuGg4tOga3v7XIweOn\neOLf37Bo7Q7aNG3I778/gB8M7Ua9elJtfbAAY4ypOgV5sCf5dKrrTOmugbdYustDBYVF/H3Vtzz5\n0TecyCtk6iXRPDCqF62aNKj2vliAMcZUXtl0164kKDzltFm6q9p9kX6AXy1JZfO+Y1xyQVvm3NSP\nXh1bBK0/FmCMMYEpN93VwEl3xd/jBJRuwyzdVY0yD+fw2w828cGGvUS0bsILdwzl2n4dceYQDh4L\nMMaYMys45TzddcZ0V7gTRAbd6r7MGAcNmgS3v3VQbn4hL/xnK89/thUR+OnVvZk2ogeNG9SMomYW\nYIwxjpxDpd892fVVmXTX1advxrfrbemuIFJVlm7cy6/f38SuIye5YWBnZl0fQ9fwmhXkLcAYUxep\nwsGtpd89OfCN01Y23RU5HJp3CG5/TYlv9h1jTkIKX2w9SN9OLXjznuFc1LNtsLt1RhZgjKkLCk7B\n7uTT9052roacA05bSbprgqW7arCjOfn86eNv+Nuqb2neqD5zx/TjtvhI6ofV3CtJCzDG1EYnDrr3\nTYpfZvz6dLqrTQ/odY2lu0JEYZHyj8SdPL5sM4dz8rgtPpKfXdOHNs0aBrtrFbIAY0yoqyjd1WWw\nk+4qnqre0l0hI+nbQ8xJSGXDrqNcGNWa2TfF079rq2B3K2CeBhgRGQ3Mx6lo+bKqzivT3hpYCPQE\ncoGpqrrRbQsHXgb6A+q2fSkic4B7APdxFmap6gfuPo/gVMUsBB5Q1WVejs+YoCgv3dWktZvumui+\nzDjE0l0haF92Lr//MI3/+3oXnVo2Zv6Ewdw8qEvQHzs+V54FGBEJA54DrgYygbUikqCqqX6bzQKS\nVXWsiPR1tx/lts0HlqrqeLdssv8rv39S1SfKnC8Wp5RyP6AL8LGI9FbVQi/GZ0y1CTTdFXkRtO1l\n6a4QdqqgkFf/u51nPtlCfqHy4yt6ct/IC2jWKDSTTV72Oh5IV9UMABFZBIwB/ANMLDAPQFXTRCRK\nRDriXM2MACa7bXlAXgXnGwMsUtVTwDYRSXf78GWVjcgYr6nCwfTSLzMe3OK0WbqrVluetp+576Wy\n7cAJrorpwKM3xBLVrlmwu3VevAwwXYGdfsuZwLAy26wDxgErRSQe6A5E4KS4soBXRWQQkAQ8qKon\n3P1miMgkIBH4maoeds+3qsz5ulbtkIypYgWnnCsS/5cZcw46bcXprsG3WbqrFtt24ASPvZfKp2n7\n6dGuGa9OuZCRfWrHLw7Bvu6aB8wXkWRgA/A1TnCpD8QBM1R1tYjMB2YCvwSeBx7DuS/zGPBHYGqg\nJxSRacA0gMjIyKobiTGBKE537fjS+bn7ayh0L87b9ITeo0+/e2Lprlrt+KkCnv00nVc+z6BR/TBm\nXd+XyRdH07B+7fk39zLA7AK6+S1HuOtKqGo2MAVAnLtX24AMnPstmaq62t30HZwAg6ruK95fRF4C\n3gv0fO7+C4AFAD6fTys3NGMCUGG6awgM+5HzqHC3YdC8fXD7a6qFqvJu8i5+90Ea+4+dYvzQCH4x\nug8dWjQOdteqnJcBZi3QS0Sicb7oJwC3+W/gPimW495juRtY4QadbBHZKSJ9VHUzzo3/VHefzqq6\nxz3EWGCj+zkBeENEnsS5yd8LWOPh+IwpLT/3u1PVl0p3DYchtzs/uwyBBrXvC8WUb0PmUeYscYp/\nDYxoxQs/HEpcZPUU/woGzwKMqhaIyP3AMpzHlBeqaoqITHfbXwBigNdERIEUnEeMi80AXnefIMvA\nvdIBHheRwTgpsu3Aj9zjpYjI2ziBqAC4z54gM546caD03F2W7jJn4RT/2syitTtp07Qhj39/IOOH\nRlRr8a9gENW6myXy+XyamJgY7G6YUKAKB7b4vXuyykl/AYQ1hM6DT78Zb+ku4/Iv/pWTV8idF0cF\nrfhXVRKRJFX1VbRdsG/yG1Mz5ec6VyT+LzOePOS0NWnjBJEhd1i6y5zVF+kHmLMkhW/2HefSC9ox\n+6bYoBb/CgYLMMZA+emuthdAn+v95u7qBSH2RrWpPjsPOcW/PtzoFP968YdDuSY2+MW/gsECjKl7\nKkp3dRkCw6affpmxWbvg9teEhLLFv352dW/uqUHFv4LBAoyp/QJNd0Ve5NxLsXSXOQdli3/dOLAz\nj9TA4l/BYAHG1D4nDpR+92RPsqW7jCc27z3Gr5acLv61aNpwhveomcW/gsECjAltqs7U9MX3Tnas\ngkNbnTZLdxmPlC3+9diYfkys4cW/gsECjAktxemu4qlWdq6Gk4edtiZtnEASN8n5aekuU8UKi5S3\nE3fyB7/iXw9d04fWIVD8KxgswJia7XhW6anqS6W7ekHfG5xUV+RwJ/1l6S7jkaRvDzE7IYWNu7JD\nsvhXMFiAMTVHhemuOBh+7+mXGZtZrtt4b192LvM+TGNxiBf/CgYLMCZ48nNh91el5+4qTnc1besE\nEUt3mSA5VVDIws+388ynWygoVO4feQH3XtEzZIt/BYP9TZnqczyrdN343clQlO+0WbrL1CCfpu1j\n7pJUth/M4aqYjvzyxhi6tw3t4l/BYAHGeKOoyEl3+b/MeCjDaStOd130Y0t3mRolI+s4j72XyvLN\nWfRo34y/TLmQK2pJ8a9gsABjqkb+ye9WZiyV7hoOQye7c3cNhvqNgtpdY/yVLf71P9fHcOfFUbWq\n+FcwWIAxlVNeuqtdb+h7o/vuyXBo29PSXaZGqkvFv4LBAoypWLnprkbOy4wX/diZaiUi3tJdJiRs\nyDzK7ISNfLXjCIMiWvHiD4cypBYX/woGCzDmu/JPwq6vSs/dlXvEabN0lwlx/sW/2jZryOPjBzI+\nrvYX/woGCzAGju8v/e7JnnWl010xN1m6y4S8fL/iXyfzCrnrkmgeuKoXLRuHdvGvmszTACMio4H5\nOCWTX1bVeWXaWwMLgZ5ALjBVVTe6beHAy0B/nPLIU1X1SxH5A3ATkAdsBaao6hERiQI2AZvdw69S\n1eleji8kFae7iqda2bEKDm9z2sIaQdc4uOi+03N3NW0T3P4aUwX+m36AX7nFvy7r5RT/uqBD3Sr+\nFQyeBRgRCQOeA64GMoG1IpKgqql+m80CklV1rIj0dbcf5bbNB5aq6ngRaQg0ddd/BDyiqgUi8nvg\nEeBht22rqg72akwhqdx0VzsnkPimui8zDrJ0l6lV/It/dWvThAU/HMrVdbT4VzB4eQUTD6SragaA\niCwCxgD+ASYWmAegqmkiEiUiHXGuZkYAk922PJwrFlT13377rwLGeziG0FNuuqsPxN58+mXGNj0s\n3WVqpZN5TvGvF/5jxb+CycsA0xXY6becCQwrs806YBywUkTige5ABFAIZAGvisggIAl4UFVPlNl/\nKvCW33K0iCQDR4FHVXVl2U6JyDRgGkBkZGQlh1ZDFBXBgc2lA4qlu0wdpqp8uHEvv/Er/jXr+hi6\nWPGvoAj2Tf55wHw3KGwAvsYJLvWBOGCGqq4WkfnATOCXxTuKyP8ABcDr7qo9QKSqHhSRocC7ItJP\nVbP9T6iqC4AFAD6fTz0dXVXLyykzd9caS3cZ49q89xhzElL4MsOKf9UUXgaYXUA3v+UId10J98t/\nCoA4SdFtQAbO/ZZMVV3tbvoOToDB3XYycCMwSlXVPdYp4JT7OUlEtgK9gcSqHli1Obav9Lsne9ZB\nUYHTZukuYwAr/lWTeRlg1gK9RCQaJ7BMAG7z38B9UizHvcdyN7DCDTrZIrJTRPqo6macG/+p7j6j\ngV8Al6tqjt+x2gOHVLVQRHoAvXCCVWj4TrrrSzi83Wmr39iZu+viGe7cXfGW7jJ1XmGR8tbanfxh\nWRpHT+Zz27BIfna1Ff+qSTwLMO5TXvcDy3AeU16oqikiMt1tfwGIAV4TEQVSgLv8DjEDeN19giwD\n90oHeBZoBHzkPglS/DjyCGCuiOQDRcB0VT3k1fjO23fSXash96jT1qy9c8/kwrudgNJ5ENS3/2mM\nKZa4/RBzljjFv+Kj2jD75lj6dbHiXzWNuBmmOsnn82liYjVl0MpLd7Xv6wSU4pvxlu4y5ozKFv+a\ndUMMNw3sbI8dVzMRSVJVX0XbBfsmf+1UVARZaaUDin+6q+tQuPgBJ6BEXGjpLmMqcKbiXz8e2ZOm\nDe0rrCazf52qkJcDu5JOB5TMNZbuMqaK+Bf/ujq2I4/eYMW/QoUFmMrIPQoZnzn3T3asgr3rS6e7\nYr9n6S5jzlPZ4l+vTY3n8t7tg90tcw4swFTGgS3w9iRLdxnjgeOnCnjm0y0s/HwbjeqH8egNMUy6\nyIp/hSILMJXRaSDc/Ynz09JdxlSJoiK3+NeHaWQdO8UPhkbwcyv+FdIswFRG/YYQUeEDFMaYAK3P\nPMKchBSn+Fe3cBZY8a9awQKMMSZoDhw/xRPLNvNWohX/qo0swBhjql1+YRF/+/Jb/vSxFf+qzSzA\nGGOq1X/TDzAnIYUt+634V21nAcYYUy12HsrhN+9vYmmKFf+qKyzAGGM8dTKvkOf/s5UX/7OVeiI8\ndE1v7r7Min/VBRUGGBGZAfxdVQ9XQ3+MMbVE2eJfNw3qwiPX9bXiX3VIIFcwHYG1IvIVsBBYpnV5\nhkxjTIXS9mbzq4TUkuJfb00bzjAr/lXnVBhgVPVREfklcA3OlPnPisjbwCuqutXrDhpjQod/8a8W\njevz2Pf6M/HCblb8q44K6B6MqqqI7AX24pQpbg28IyIfqeovvOygMabmK1v86/Zh3fnp1b2t+Fcd\nF8g9mAeBScAB4GXg56qaLyL1gC041SWNMXVU4vZDzE5IIWV3NvHRbZhzUz9iu7QMdrdMDRDIdWsb\nYJyqXquq/1DVfABVLQJuLG9HERktIptFJF1EZp6hvbWILBaR9SKyRkT6+7WFi8g7IpImIptE5CJ3\nfRsR+UhEtrg/W/vt84h7rs0icm2AfwfGmErYezSX/7foa8a/8CWHTuTxzMQhvDVtuAUXUyKQFNmH\nQEnpYRFpCcSo6mpV3XS2nUQkDHgOuBrIxHlQIEFVU/02mwUkq+pYEenrbj/KbZsPLFXV8W7Z5Kbu\n+pnAJ6o6zw1aM4GHRSQWmAD0A7oAH4tIb1UtDGCMxpgAnSoo5JXPt/Hsp+kUFCkzrryAe6+w4l/m\nuwL5L+J5IM5v+fgZ1p1JPJCuqhkAIrIIGAP4B5hYYB6AqqaJSJSIdARygRHAZLctD8hz9xkDXOF+\nfg34DHjYXb9IVU8B20Qk3e3DlwGM0RgTgE827WPue6l8ezCHa2I78ugNsUS2bVrxjqZOCiTAiP9j\nyapaJCKB7NcV2Om3nAkMK7PNOmAcsFJE4oHuQARQCGQBr4rIICAJeFBVTwAdVXWPu/9enMeoi8+3\nqsz5ugbQT2NMBTKyjjP3vVQ+c4t//XVqPCOs+JepQCD3YDJE5AERaeD+eRDIqKLzzwPCRSQZmAF8\njRNc6uNcIT2vqkOAEzipsFLcwHdO7+SIyDQRSRSRxKysrPPtvzG12rHcfH73wSaufWoFidsP8+gN\nMSx9cIQFFxOQQK5EpgNPA4/ifJl/AkwLYL9dQDe/5Qh3XQlVzcZ5twZxJiTahhO8mgKZqrra3fQd\nTgeYfSLSWVX3iEhnYH+g53PPuQBYAODz+eyFUWPOoKhIWfz1LuYtteJfpvICedFyP87N83O1Fugl\nItE4X/QTgNv8NxCRcCDHvcdyN7DCDTrZIrJTRPqo6macG//F924SgDtxrn7uBP7lt/4NEXkS5yZ/\nL2BNJfptTJ22PvMIsxNS+Not/vXSJB+Du4UHu1smBAXyHkxj4C6cp7NKfn1R1anl7aeqBSJyP7AM\nCAMWqmqKiEx3218AYoDXRESBFPc8xWYAr7tPkGXgXungBJa3ReQu4FvgFvd4Ke4MA6k4L4PeZ0+Q\nGRO4A8dP8Yelm3k7aSdtmzXiD+MH8n0r/mXOg1Q0rZiI/ANIw7n6mAvcDmxS1Qe97563fD6fJiYm\nBrsbxgRV2eJfUy6JYsYoK/5lzk5EklS1wrrxgdyDuUBVfyAiY1T1NRF5A1h5/l00xgTb51sO8Ksl\n/sW/+nFBh+bB7papJQIJMPnuzyPum/Z7gQ7edckY4zX/4l+RbZry0iQfV8V0sOJfpkoFEmAWuNOx\nPIpzI7058EtPe2WM8UTZ4l8/v7YPd10abcW/jCfKDTDuhJbZbrGxFUCPaumVMaZKqSofbNjLb95P\nZffRXG4e1IVHru9L51ZW/Mt4p9wA4761/wvg7WrqjzGmiqXtzWZOQgqrMg4R07klf7p1sBX/MtUi\nkBTZxyLyEPAWzhv1AKjqobPvYowJtiM5efzpI6f4V8smDfj19/ozMT6SMHvs2FSTQALMre7P+/zW\nKZYuM6ZGKixSFq3dwRPLNnP0ZD53DHeKf4U3teJfpnoF8iZ/dHV0xBhz/vyLfw2LbsOcm/sR09nq\ns5jgCORN/klnWq+qf6367hhjKmPv0Vx+9+Em/pW8m86tGvPMxCHcOLCzPXZsgiqQFNmFfp8b48wL\n9hVgAcaYIDtVUMjLK7fx3HIr/mVqnkBSZDP8l90JKhd51iNjTIVUlU/T9lvxL1OjVebXnBOA3Zcx\nJki2Zh3nMbf4V08r/mVqsEDuwSzhdFGvejhlju29GGOq2bHcfJ79NJ2F/91G4/phPHpDDHdeHEWD\nsEDqBhpT/QK5gnnC73MB8K2qZnrUH2NMGWWLf93ii+Dn1/alfYtGwe6aMeUKJMDsAPaoai6AiDQR\nkShV3e5pz4wxrNt5hDlLnOJfg634lwkxgQSYfwAX+y0XuusuPPPmxpjzVbb41xM/GMS4IV2t+JcJ\nKYEEmPpuSWMAVDXPrTJZIREZDczHqWj5sqrOK9PeGlgI9ARygamqutFt2w4cwwloBcXFbUTkLaCP\ne4hw4IiqDhaRKGATsNltW6Wq0wPppzE1RX5hEX/98luecot/3XNZD2ZceQEtrPiXCUGBBJgsEblZ\nVRMARGQMcKCinUQkDHgOuBrIBNaKSIKqpvptNgtIVtWxItLX3X6UX/tIVS11LlUtnroGEfkjcNSv\neauqDg5gTMbUOJ9vOcCcJSmk7z/OiN7t+d8bY634lwlpgQSY6cDrIvKsu5wJnPHt/jLigXRVzQAQ\nkUXAGMA/wMQC8wBUNU1EokSko6ruq+jg4ryifAtwZQB9MabG2nkoh1+/n8qylH1EtmnKy5N8jLLi\nX6YWCORFy63AcBFp7i4fD/DYXYGdfsuZwLAy26wDxgErRSQe6A5EAPtwHo3+WEQKgRdVdUGZfS8D\n9qnqFr910SKSjHNV86iqWmlnU2OdzCvk+c/SeWFFBmFW/MvUQoG8B/Nb4HFVPeIutwZ+pqqPVsH5\n5wHz3aCwAfga554LwKWquktEOgAfiUiaqq7w23ci8Kbf8h4gUlUPishQ4F0R6aeq2WXGMw2YBhAZ\nGVkFQzDm3JQt/jVmcBdmXmfFv0ztE0iK7DpVnVW8oKqHReR6nBLK5dkFdPNbjnDXlXC//KdAScpr\nG5Dhtu1yf+4XkcU4KbcV7rb1ca58hvod6xRwyv2cJCJbgd5AYplzLgAWAPh8PsWYalS2+NdTE4YQ\nH90m2N0yxhOBBJgwEWnkfoEjIk2AQN7wWgv0EpFonMAyAbjNfwN3XrMc9ym1u4EVqpotIs2Aeqp6\nzP18DTDXb9ergDT/Fz5FpD1wSFULRaQH0As3WBkTbEdy8njyo2/4uxX/MnVIIAHmdeATEXkVEGAy\n8FpFO6lqgYjcDyzDeUx5oaqmiMh0t/0FIAZ4TUQUSAHucnfvCCx2b3LWB95Q1aV+h59A6fQYwAhg\nrojkA0XAdKu6aYLNin+ZukxUK84Sue+zXIVz4z0b6KSq95W/V83n8/k0MTGx4g2NqYS12w8x+18p\npO6x4l+mdhGRpOJ3E8sT6GzKxU91/QDnPsk/z6NvxtRqZYt/PXvbEG4YYMW/TN1z1gAjIr1xntSa\niPNi5Vs4Vzwjq6lvxoSU3PxCXvn8dPGvB668gOlW/MvUYeX9l58GrARuVNV0ABH5SbX0ypgQoqp8\nsmk/j73vFP+6tp9T/KtbGyv+Zeq28gLMOJyb6ctFZClOFUu7xjfGz9as48xdksp/vsnigg7N+dtd\n8VzWy4p/GQPlBBhVfRfnZcVmOFO8/D+gg4g8DyxW1X9XUx+NqXGO5ebzzKfpLPx8G00ahPHLG2OZ\ndFF3K/5ljJ9Apoo5AbwBvOG+xf8D4GHAAoypc4qKlP/7ehfzPkzj4IlT3DK0Gz8f3Yd2za34lzFl\nndPdR1U9jPMWfNl5wYyp9dbtPMLshBSSdzrFv16508cgK/5lzFnZ4y3GVCDr2Cn+sCyNtxMzade8\nEX/8wSDGWvEvYypkAcaYsygp/vXRN+QWFPKjET2434p/GRMwCzDGnMHKLVn8aklqSfGv2TfF0rO9\nFf8y5lxYgDHGz46DTvGvf6da8S9jzpcFGGOw4l/GeMECjKnTVJX3N+zht+9vsuJfxlQxCzCmztq0\nxyn+tXrbIWI7t2T+xCFcGGXFv4ypKhZgTJ3jX/yrVZMG/GZsfyZcaMW/jKlqFmBMnVFYpLy5ZgdP\n/Hsz2Sfz+eHw7vzEin8Z4xkLMKZOWLPtEHMSnOJfw3u0YfZNVvzLGK95OjOfiIwWkc0iki4iM8/Q\n3lpEFovIehFZIyL9/dq2i8gGEUkWkUS/9XNEZJe7PllErvdre8Q912YRudbLsZnQsOfoSR5482tu\nefFLjuTk8dxtcbx5z3ALLsZUA8+uYEQkDHgOuBrIBNaKSIKqpvptNgtIVtWxItLX3X6UX/tIVT1w\nhsP/SVWfKHO+WJzyAv2ALsDHItJbVQurblQmVBQX/3r203QKVXlgVC/uvbwnTRraY8fGVBcvU2Tx\nQLqqZgCIyCKcaf/9A0wsMA9AVdNEJEpEOqrqvkqcbwywSFVPAdtEJN3tw5fnMwgTWlSVjzft57H3\nUtlxyIpyEvrBAAAXIElEQVR/GRNMXqbIugI7/ZYz3XX+1uEUNkNE4oHuQITbpjhXIUkiMq3MfjPc\ntNpCt4RAoOcztdjWrONMfnUt9/w1kYb16/H3u4bx4g99FlyMCZJg3+SfB8wXkWRgA/A1UJzSulRV\nd4lIB+AjEUlT1RXA88BjOAHoMeCPwNRAT+gGq2kAkZGRVTYQEzylin81DON/b4zlh1b8y5ig8zLA\n7AK6+S1HuOtKqGo2MAVAnMmetgEZbtsu9+d+EVmMk+5a4Z8+E5GXgPcCPZ97vJJ6Nj6fTys/PBNs\nVvzLmJrNywCzFuglItE4X/QTgNv8NxCRcCBHVfOAu3ECSLZbprmeqh5zP18DzHX36ayqe9xDjAU2\nup8TcKpuPolzk78XsMbD8Zkg8i/+NSTSin8ZUxN5FmBUtUBE7geWAWHAQlVNEZHpbvsLQAzwmogo\nkALc5e7eEVjszmBbH3hDVZe6bY+LyGCcFNl24Efu8VJE5G2chwgKgPvsCbLax7/4V/sWVvzLmJpM\nVOtulsjn82liYmLFG5qgyy8s4rUvtjP/4y3kFhQy9ZJoK/5lTJCISJKq+iraLtg3+Y2pkH/xr8t7\nt+d/rfiXMSHBAoypsfyLf3Vv25RX7vRxZV8r/mVMqLAAY2qcnLwCnv9sKy+uyKB+PeEXo53iX43q\n21v4xoQSCzCmxlBV3lu/h99+sIk9R3P53uAuzLwuhk6tGge7a8aYSrAAY2qE1N3ZzFmSwppth+jX\npSVPW/EvY0KeBRgTVIdPOMW/Xl/tFP/67dgB3HphNyv+ZUwtYAHGBEVhkfLGmh380Yp/GVNrWYAx\n1W7NtkPMTkhhk1v8a87N/ejbyeqzGFPbWIAx1WbP0ZP87oM0Etbtpmt4E/58exzX9e9kjx0bU0tZ\ngDGe8y/+VaTKg6N6Md2KfxlT61mAMZ4pW/xrdL9O/M8NMVafxZg6wgKM8UT6/uPMfS+VFd9k0atD\nc/5+1zAu7dUu2N0yxlQjCzCmSh3LzefpT7bw6n+3W/EvY+o4CzCmShQVKf/8KpPfL93MwROnuNXX\njYeuteJfxtRlFmDMeUt2i3+tc4t/LZzsY2CEFf8ypq6zAGMqLevYKR5fmsY/kpziX0/eMojvDbbi\nX8YYh6cBRkRGA/NxKlq+rKrzyrS3BhYCPYFcYKqqbnTbtgPHgEKgoLi4jYj8AbgJyAO2AlNU9YiI\nRAGbgM3u4Vep6nQvx1dXlS3+9aPLezDjyl40b2S/rxhjTvPsG0FEwoDngKuBTGCtiCSoaqrfZrOA\nZFUdKyJ93e1H+bWPVNUDZQ79EfCIW5L598AjwMNu21ZVHezFeIxjxTdZ/GpJCluzTnBFn/b8742x\n9LDiX8aYM/DyV854IF1VMwBEZBEwBvAPMLHAPABVTRORKBHpqKr7znZQVf233+IqYHyV99x8x46D\nOTz2fiofpe4jqm1TFk72cWXfjsHuljGmBvMywHQFdvotZwLDymyzDhgHrBSReKA7EAHsAxT4WEQK\ngRdVdcEZzjEVeMtvOVpEkoGjwKOqurJKRlKH5eQV8OflW1mw0op/GWPOTbCT5vOA+W5Q2AB8jXPP\nBeBSVd0lIh2Aj0QkTVVXFO8oIv8DFACvu6v2AJGqelBEhgLvikg/Vc32P6GITAOmAURGRno5tpBm\nxb+MMefLywCzC+jmtxzhrivhfvlPARBnxsNtQIbbtsv9uV9EFuOk3Fa4204GbgRGqaq6250CTrmf\nk0RkK9AbSCxzzgXAAgCfz6dVNtpapGzxr2cmDsFnxb+MMefIywCzFuglItE4gWUCcJv/BiISDuSo\nah5wN7BCVbNFpBlQT1WPuZ+vAea6+4wGfgFcrqo5fsdqDxxS1UIR6QH0wg1WJjD+xb/Cmzbkd+MG\ncIvPin8ZYyrHswDjPuV1P7AM5zHlhaqaIiLT3fYXgBjgNRFRIAW4y929I7DYnca9PvCGqi51254F\nGuGkzeD048gjgLkikg8UAdNV9ZBX46tN/It/HcstYNJFUfzkqt60atog2F0zxoQwcTNMdZLP59PE\nxMSKN6zFVmccZM6SVDbtyeaiHm2ZfXOsFf8yxpRLRJKK300sT7Bv8psg2X3kJL/7MI0lVvzLGOMR\nCzB1TG5+IS+vzOC55Vut+JcxxlMWYOoIVeWj1H089n4qOw+d5Lr+nZh1vRX/MrVHfn4+mZmZ5Obm\nBrsrtUbjxo2JiIigQYPK3Y+1AFMHlC3+9frdw7jkAiv+ZWqXzMxMWrRoQVRUlKV6q4CqcvDgQTIz\nM4mOjq7UMSzA1GLZufk8/fEW/vKFU/xr9k2x3DHcin+Z2ik3N9eCSxUSEdq2bUtWVlalj2EBphYq\nKlLe+SqTx5emcfBEHhMu7MZD1/ShrRX/MrWcBZeqdb5/nxZgahn/4l9xkeG8OjmeARGtgt0tY2q9\ngwcPMmqUMxn83r17CQsLo3379gCsWbOGhg0bVniMKVOmMHPmTPr06XPWbZ577jnCw8O5/fbbq6bj\nHrIAU0v4F//q0KIRf7rVKf5lv9EZUz3atm1LcnIyAHPmzKF58+Y89NBDpbZRVVSVevXOnKZ+9dVX\nKzzPfffdd/6drSaWjA9xeQVFvLwygyuf+Ix3k3fxo8t78OlDVzB2SIQFF2NqgPT0dGJjY7n99tvp\n168fe/bsYdq0afh8Pvr168fcuXNLtr300ktJTk6moKCA8PBwZs6cyaBBg7jooovYv38/AI8++ihP\nPfVUyfYzZ84kPj6ePn368MUXXwBw4sQJvv/97xMbG8v48ePx+Xwlwa862RVMCPMv/jWyT3t+acW/\njAHgV0tSSN2dXfGG5yC2S0tm39SvUvumpaXx17/+FZ/Pefl93rx5tGnThoKCAkaOHMn48eOJjY0t\ntc/Ro0e5/PLLmTdvHj/96U9ZuHAhM2fO/M6xVZU1a9aQkJDA3LlzWbp0Kc888wydOnXin//8J+vW\nrSMuLq5S/T5fdgUTgnYczOGevyYyaeEaCouUhZN9vDol3oKLMTVUz549S4ILwJtvvklcXBxxcXFs\n2rSJ1NTU7+zTpEkTrrvuOgCGDh3K9u3bz3jscePGfWebzz//nAkTJgAwaNAg+vWrXGA8X3YFE0LK\nFv96eHRfpl4aZcW/jCmjslcaXmnWrFnJ5y1btjB//nzWrFlDeHg4d9xxxxlfDvV/KCAsLIyCgoIz\nHrtRo0YVbhMsdgUTAlSVhHW7GfXH//Ds8nRuGNCZ5Q9dwb1X9LTgYkyIyc7OpkWLFrRs2ZI9e/aw\nbNmyKj/HJZdcwttvvw3Ahg0bzniFVB3sCqaGS92dzZyEFNZsP0T/rlb8y5hQFxcXR2xsLH379qV7\n9+5ccsklVX6OGTNmMGnSJGJjY0v+tGpV/a8r2HT9NXS6/sMn8vjjR5t5Y/UOwps25OfX9rHiX8aU\nY9OmTcTExAS7GzVCQUEBBQUFNG7cmC1btnDNNdewZcsW6tc/92uKM/292nT9IaqwSHlj9bc88e9v\nOH7Kin8ZY87d8ePHGTVqFAUFBagqL774YqWCy/myAFODlC3+NefmfvTp1CLY3TLGhJjw8HCSkpKC\n3Q1vb/KLyGgR2Swi6SLynQe4RaS1iCwWkfUiskZE+vu1bReRDSKSLCKJfuvbiMhHIrLF/dnar+0R\n91ybReRaL8dWlXYfOcmMN7/m1gWryD6Zz/O3x/HGPcMsuBhjQppnVzAiEgY8B1wNZAJrRSRBVf0f\nZ5gFJKvqWBHp624/yq99pKoeKHPomcAnqjrPDVozgYdFJBaYAPQDugAfi0hvVS30ZIBVIDe/kJdW\nZPDnz5ziX//vql78aIQV/zLG1A5epsjigXRVzQAQkUXAGMA/wMQC8wBUNU1EokSko6ruK+e4Y4Ar\n3M+vAZ8BD7vrF6nqKWCbiKS7ffiyykZURcoW/7p+gFP8K6K1Ff8yxtQeXgaYrsBOv+VMYFiZbdYB\n44CVIhIPdAcigH2A4lyFFAIvquoCd5+OqrrH/bwX6Oh3vlVlzte1bKdEZBowDSAyMrJyIzsP6fuP\n8aslqazccoDeHZvzxt3DuNiKfxljaqFgv2g5DwgXkWRgBvA1UJzSulRVBwPXAfeJyIiyO6vzjPU5\nPWetqgtU1aeqvuKptKtDdm4+v34vldFPrXSm1L8plvcfuMyCizG1xMiRI7/z0uRTTz3Fvffee9Z9\nmjd3pnfavXs348ePP+M2V1xxBRW9TvHUU0+Rk5NTsnz99ddz5MiRQLvuGS8DzC6gm99yhLuuhKpm\nq+oUN5BMAtoDGW7bLvfnfmAxTroLYJ+IdAZwf+4P9HzBUFSkvJ24kyuf+IxX/ruNH/gi+OyhK5hy\nSbRVljSmFpk4cSKLFi0qtW7RokVMnDixwn27dOnCO++8U+lzlw0wH3zwAeHh4ZU+XlXx8htuLdBL\nRKJFpCHODfgE/w1EJNxtA7gbWKGq2SLSTERauNs0A64BNrrbJQB3up/vBP7lt36CiDQSkWigF7DG\no7EF5Osdhxn75//yi3fWE9mmKQn3Xcrvxg20ypLG1ELjx4/n/fffJy8vD4Dt27eze/duhgwZwqhR\no4iLi2PAgAH861//+s6+27dvp39/5yHakydPMmHCBGJiYhg7diwnT54s2e7ee+8tmeZ/9uzZADz9\n9NPs3r2bkSNHMnLkSACioqI4cMB5PurJJ5+kf//+9O/fv2Sa/+3btxMTE8M999xDv379uOaaa0qd\np6p4dg9GVQtE5H5gGRAGLFTVFBGZ7ra/AMQAr4mIAinAXe7uHYHFbj2T+sAbqrrUbZsHvC0idwHf\nAre4x0sRkbdxHiIoAO4L1hNk+4/l8vjSzbxjxb+MCY4PZ8LeDVV7zE4D4Lp5Z21u06YN8fHxfPjh\nh4wZM4ZFixZxyy230KRJExYvXkzLli05cOAAw4cP5+abbz7r98Hzzz9P06ZN2bRpE+vXry811f5v\nfvMb2rRpQ2FhIaNGjWL9+vU88MADPPnkkyxfvpx27Uqn3JOSknj11VdZvXo1qsqwYcO4/PLLad26\nNVu2bOHNN9/kpZde4pZbbuGf//wnd9xxR9X8Xbk8fdFSVT8APiiz7gW/z18Cvc+wXwYw6CzHPEjp\nR5n9234D/OY8unxe8gqKeO2L7cz/ZAunCgqZfnlP7r/yApo3svdZjakLitNkxQHmlVdeQVWZNWsW\nK1asoF69euzatYt9+/bRqVOnMx5jxYoVPPDAAwAMHDiQgQMHlrS9/fbbLFiwgIKCAvbs2UNqamqp\n9rI+//xzxo4dWzKb87hx41i5ciU333wz0dHRDB48GCi/HMD5sG++KvIft/hXRtYJruzbgV/eGEt0\nu2YV72iMqXrlXGl4acyYMfzkJz/hq6++Iicnh6FDh/KXv/yFrKwskpKSaNCgAVFRUWecnr8i27Zt\n44knnmDt2rW0bt2ayZMnV+o4xYqn+Qdnqn8vUmR2l/k8fXvwBHe/lsidC9egCgsn+1g4+UILLsbU\nQc2bN2fkyJFMnTq15Ob+0aNH6dChAw0aNGD58uV8++235R5jxIgRvPHGGwBs3LiR9evXA840/82a\nNaNVq1bs27ePDz/8sGSfFi1acOzYse8c67LLLuPdd98lJyeHEydOsHjxYi677LKqGm6F7AqmknLy\nCnhueTovrdhGgzBh5nV9mXKJFf8ypq6bOHEiY8eOLXmi7Pbbb+emm25iwIAB+Hw++vbtW+7+9957\nL1OmTCEmJoaYmBiGDh0KOJUphwwZQt++fenWrVupaf6nTZvG6NGj6dKlC8uXLy9ZHxcXx+TJk4mP\ndx7CvfvuuxkyZIgn6bAzsen6KzFd/7qdR/jR35LYm53LuCFdefi6vnRs2diDHhpjAmXT9XvDpuuv\nZt3bNqVXx+Y8d/sQhna34l/GGHMmFmAqIbxpQ/52V9lZb4wxxvizm/zGGGM8YQHGGFNr1OV7yl44\n379PCzDGmFqhcePGHDx40IJMFVFVDh48SOPGlX+Aye7BGGNqhYiICDIzM8nKygp2V2qNxo0bExER\nUen9LcAYY2qFBg0aEB0dHexuGD+WIjPGGOMJCzDGGGM8YQHGGGOMJ+r0VDEikoVTU6ay2gEHqqg7\noaCujRdszHWFjfncdFfVCmvO1+kAc75EJDGQ+Xhqi7o2XrAx1xU2Zm9YiswYY4wnLMAYY4zxhAWY\n87Mg2B2oZnVtvGBjritszB6wezDGGGM8YVcwxhhjPGEBpgIiMlpENotIuojMPEO7iMjTbvt6EYkL\nRj+rUgBjvt0d6wYR+UJEBgWjn1WpojH7bXehiBSIyPjq7J8XAhmziFwhIskikiIi/6nuPla1AP7b\nbiUiS0RknTvmKcHoZ1URkYUisl9ENp6l3dvvL1W1P2f5A4QBW4EeQENgHRBbZpvrgQ8BAYYDq4Pd\n72oY88VAa/fzdXVhzH7bfQp8AIwPdr+r4d85HEgFIt3lDsHudzWMeRbwe/dze+AQ0DDYfT+PMY8A\n4oCNZ2n39PvLrmDKFw+kq2qGquYBi4AxZbYZA/xVHauAcBHpXN0drUIVjllVv1DVw+7iKqDy063W\nDIH8OwPMAP4J7K/OznkkkDHfBvyfqu4AUNVQH3cgY1aghYgI0BwnwBRUbzerjqquwBnD2Xj6/WUB\npnxdgZ1+y5nuunPdJpSc63juwvkNKJRVOGYR6QqMBZ6vxn55KZB/595AaxH5TESSRGRStfXOG4GM\n+VkgBtgNbAAeVNWi6uleUHj6/WXT9ZtKE5GROAHm0mD3pRo8BTysqkXOL7d1Qn1gKDAKaAJ8KSKr\nVPWb4HbLU9cCycCVQE/gIxFZqarZwe1WaLIAU75dQDe/5Qh33bluE0oCGo+IDAReBq5T1YPV1Dev\nBDJmH7DIDS7tgOtFpEBV362eLla5QMacCRxU1RPACRFZAQwCQjXABDLmKcA8dW5QpIvINqAvsKZ6\nuljtPP3+shRZ+dYCvUQkWkQaAhOAhDLbJACT3KcxhgNHVXVPdXe0ClU4ZhGJBP4P+GEt+W22wjGr\narSqRqlqFPAO8OMQDi4Q2H/b/wIuFZH6ItIUGAZsquZ+VqVAxrwD54oNEekI9AEyqrWX1cvT7y+7\ngimHqhaIyP3AMpwnUBaqaoqITHfbX8B5ouh6IB3IwfkNKGQFOOb/BdoCf3Z/oy/QEJ4oMMAx1yqB\njFlVN4nIUmA9UAS8rKpnfNw1FAT47/wY8BcR2YDzZNXDqhqysyyLyJvAFUA7EckEZgMNoHq+v+xN\nfmOMMZ6wFJkxxhhPWIAxxhjjCQswxhhjPGEBxhhjjCcswBhjjPGEBRhjPCAihe4sxMV/zjpDcyWO\nHXW22XGNqUnsPRhjvHFSVQcHuxPGBJNdwRhTjURku4g87tbSWSMiF7jro0TkU7cmxyfubAmISEcR\nWezWJ1knIhe7hwoTkZfcmiX/FpEm7vYPiEiqe5xFQRqmMYAFGGO80qRMiuxWv7ajqjoAZ+bep9x1\nzwCvqepA4HXgaXf908B/VHUQTl2PFHd9L+A5Ve0HHAG+766fCQxxjzPdq8EZEwh7k98YD4jIcVVt\nfob124ErVTVDRBoAe1W1rYgcADqrar67fo+qthORLCBCVU/5HSMK+EhVe7nLDwMNVPXX7tQux4F3\ngXdV9bjHQzXmrOwKxpjqp2f5fC5O+X0u5PT91BuA53CudtaKiN1nNUFjAcaY6ner388v3c9f4Mzu\nC3A7sNL9/AlwL4CIhIlIq7MdVETqAd1UdTnwMNAKpyqjMUFhv90Y440mIpLst7xUVYsfVW4tIutx\nrkImuutmAK+KyM+BLE7PavsgsEBE7sK5UrkXONt06mHA390gJMDTqnqkykZkzDmyezDGVCP3Howv\nlKeANyZQliIzxhjjCbuCMcYY4wm7gjHGGOMJCzDGGGM8YQHGGGOMJyzAGGOM8YQFGGOMMZ6wAGOM\nMcYT/x9iLAkO3Wk5eQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdb308c6eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure()\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.plot(hist.history['loss'])\n",
    "plt.plot(hist.history['val_loss'])\n",
    "plt.legend(['Training', 'Validation'])\n",
    "\n",
    "plt.figure()\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.plot(hist.history['acc'])\n",
    "plt.plot(hist.history['val_acc'])\n",
    "plt.legend(['Training', 'Validation'], loc='lower right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 4: Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available Metrics in Model: ['loss', 'acc']\n"
     ]
    }
   ],
   "source": [
    "print('Available Metrics in Model: {}'.format(model.metrics_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.0933376350194\n",
      "Test Accuracy: 0.9685\n"
     ]
    }
   ],
   "source": [
    "# Evaluating the model on the test data    \n",
    "loss, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test Loss:', loss)\n",
    "print('Test Accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Let's plot our model Predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6IAAABYCAYAAADvGUplAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcTfX/wPHXnX2zjd1YZhhjLFkaREqoSCEqla8l0Upp\n1aqkVZsQKpX6tYgiJCUlosigEDPWmbHvWSbGmLlzf3+875lxxwyz3eXc+34+Hh5z59xz7nw+zrln\n+bw/n/fHYrPZUEoppZRSSimlXMXP3QVQSimllFJKKeVb9EFUKaWUUkoppZRLOf1BNDQ09IDFYrEV\n919oaOgBZ5fNGbS+3l1nX6sv6DGt9dX6mrnOvlZf0GNa66v19cX6mrXOvlbf/CzOHiNqsVhsJfkb\nFosFm81mcUKRnErrW6xtTVdnX6sv6DFdjO20vibga99hX6sv6DFdjO20viag9S3Wtqars6/VNz/t\nmquUUkoppZRSyqX0QVQppZRSSimllEu55UF04cKFNGrUiNjYWMaOHeuOIrjUkCFDqFatGs2aNXN3\nUVzG1/ax1te7+Vp9wffq7Ev11WuSd+9f8L366jHt/ftY6+ulbDabU//Jn8iTnZ1tq1+/vm3Hjh22\nzMxMW/PmzW2bNm2y5Wffzunlc3Z9bTab7bfffrOtXbvW1rRp0/Pe88b6evM+9rX6FlRnra9317eo\ndfa1+pq1ziW9Jpm1vgXV2Zv3r9ZX6H2WnqPNWmdfq2/+fy6PiCYmJhIbG0v9+vUJCgri9ttvZ968\nea4uhkt17NiRyMhIdxfDZXxtH2t9tb7extfq7Gv11WuSd+9fX6sv6DHt7ftY6+u99XX5g+jevXup\nU6dO7u+1a9dm7969ri6GciJf28daX62vt/G1OvtafX2Nr+1fX6uvL/K1faz19d76arIipZRSSiml\nlFIu5fIH0aioKHbv3p37+549e4iKinJ1MZQT+do+1vpqfb2Nr9XZ1+rra3xt//pafX2Rr+1jra8X\n19fVg3CzsrJsMTExtpSUlNwBuBs3bvSaAbj562tITU31mUH03ryPfa2+BdVZ6+vd9S1qnX2tvmat\nc0mvSWatb0F19ub9q/XNo/dZeo42Y519rb75/wU45en2AgICApg0aRLdunXDarUyZMgQmjZtWuC6\n1/r1tbm4eEX2c843lqKu269fP5YuXcqRI0eoXbs2Y8aMYejQoeet5y319YZ97Gv1haLXWevr3fWF\notfZ1+oLnltnZ1yTwDvq6w37F/ScdSF6n6XnaIOn1tnX6gtFq7PF/jTuNBaLxVaSv2GxWLjGcosT\nSlQ2CvvP1foWnSfX2dfqC3pMG7S+ReNr9bVv67F11vrm0WO6aLS+nkXrK3ztnOWt9YWiPYhqsiKl\nlFJKKaWUUi6lD6JKKaWUUkoppVzK5WNEVfGlvdweAGuIhO6rNj0MwMoWs3PXafDrnQCUSwwFoPrE\nFa4solJKKaWUKfmFhQGQsCIdgNFV1wHQNekmgq7d6bZyKeXtNCKqlFJKKaWUUsqlNCLqwY4taAjA\nxpaTCnw/65yxzZs7fwTAl61rAvD1z1cBYE3e5sQSup8lQbKILfjucwAuef8BAOq8ZO6IsH/FCgBs\nmVQ/d9+OOpQAwD/94wCwJm11T+GUUqqYAmpUB+Bsw1rnvRe4dS8AW56uD0DFJMlvEZl8BgC/5X+7\nooiqAGd6tgUg9Me/ALC1bkJqr3AAruzyDwDLf73EYZuaK60AhMxPdFUxS8yIhG6d2giAuVWnApBj\nf3/3+po0QCOiyntsf6cdO257H4BBOzsCcLD9SbeVx+kPoiEhIQctFkv14m5XPrgCnHVGiZxL61t0\nZqyzr9UX9JguKq2vOfjad9jX6gt6TBeV1tcctL5FZ8Y6+1p983P6g2hGRkYN43Wx5rox6X9sWdX3\n2IKG/NFyRoGrvn9cWo3HrbyW6HoyXnRRk28B6F9uPwCvDK4CQP0nnRsRLXF9oUz28aE25QHIRlpg\nw/Y5dzqic+sLzjumc2JqA/BPpw9yI98vV1sLQIs+lwNQx0URUWd8h62dLwXggalfA/Bew9hilyv9\ntnZUXHdEPm/L9mJvXxh3n7OOD5Ix4avGvgdAk8nDAKj7eiK27Oyy+SPnKMv6BtSrA0C1mccB+G1t\nEwDip8jv1k1bSlRG/6pVATjaXY6TSjPt0ZnMzGJ/lrvPWa7mrvqeGNAOgKPXS1TzqVYLARhU/ofz\n1v34RF0Abio3B4BKfUMc3u8RlVCsv+3u77CrlWV9/atUBsA6U/JNzGg4DoCD1kAAKvgtpW5AmONG\ndyxz+PXQgNMA7JsYxL2vPgRA5Q9XFrlYF1OW9U15tgUASZ0nAtA/pTsAR1+JAaDBwj9LXM6yosdz\nMRSxzkYPjRMdotl7rfyJ1F4SDc+yyf1kh3W3A3B4dyUAmow9QHbarmIVpyhcdV9p6NAuKff1Z/Xk\nu3tln3sBCJuzqvgfWEo6RlQppZRSSimllEvpGFEPk321tPz+2mIyIC2Q44/JmMAlt7WWlfYdAiDu\n2Br8QqTl+NVVMkbjmSoyZiO7UtlHTjzRsebScrUnWyIjlT8uu1ZXdwioI5HQmKllF+HzRDu7BQMQ\n6f9fiT/jwA1nyRoobWmRPcqkWG4VECVj5156/iOH5UnDpwDQfeKV2NLTXV6uogqoUZ0Xl0om70aB\nMsKqy1Fp6LVuKlnPDCMS2v93iYC2C5GI2fB/pPWWvzeVuLzOYESTtrwjEb5ODbex96osoGTRW7Pw\na9EYgM0PhrO863gAqvqvlveK0N49tIIRZQi54HrK+bZOkGN3S/zH9iUS/azmL79NOR7HX+myzp5T\nFR229bfI935Bo/m528wc9SYA9yVL/ga/39c5rewlcbaa473ShuWSmyNmobnvJZQjS7Dcc6SMkd5Y\nk26R6+xVoadz18myybkqxz5CeHnL6fJGS/uPykOo29cVpXUuIwp6rn0dZVx+7BxXl0YjokoppZRS\nSimlXMxjI6JH75ZxUnUHSmRo86HqnM2UCGHUV/IzbI9EU3LWJRXwCeb0X1QQIK3IRiR0aS+JdlpT\nzh9ftX1MKwCmR75tXyKtPrUXencbg62DNFEt7yHjV65a9iAAsZgzu+Ku52XcZ8J1ciy/UXN5oetG\nXC7jgnc/J9tU2SAtuqHzPD9DoSVQju8uXUrfKl7u7xBuHfobAEsqSiTZevxEqT/XXQ51qwdA17As\nh+WXrrkNgKr/eWaW5IDaUQBUmHma5kESNmn0y30ANLzjr1J9dvLL0QDcGiHjCy8d/wQAtf72rKzY\nhx6Q7+Lohz4D4IawRbnv9a7SE4DsvftcXzAXORVTDoCt3d8DQou8nZHv4MudbS64XgU8t4eIX8sm\nnKkhWWTTektU4Za2Eg3Ossn3Ycnnknm25m8nsHlYFP9ctvYtmHn5B/bf5PZwYYZERMeOvAOAcpuO\nwOF/AfA7tttxez+pb9zbMq496dZ3aRAYAUDGKMnKWWGwjM3LPnDQSbUonsAIGWSXniM/6/7svT0X\nLsTaSSKFAc8fZH6j7wAItMj+zD9msvKzcg9uSdvL0Z6SByBy7kYAcjy0186ukfZZBwZOKHSdO3de\nDcDH9X4u8P11l0+jFxc+V5lV7CPuGwvt3U8rSimllFJKKaU8jsdGRJ8YKX2zbw4/JgsanPNmJ/mR\nli19uycc7lyiv5F4SCIQ4W/LnI0Bi9eW6HPKUsXPZFzCLWsGYDkmLYjZ+9MKXf+u638BIMIv2NlF\n8yj/NpFW95r+0lobNSvQncUptQ33vgvktTxeyNIWX8oLSfbHnFMyd+y09N4E/Or+Y/hC0vtIq+vE\nKKlv47kybqghxc/UllnJxohKmwFYWk7GqGHCiKgxj123Eb8X+H7wDMnYh825GaFL6lgHyZQ7N3py\n7rLGo2Qce2lGqtvat2B7D4nOXPWPDMypM03298W/Ja7hHycXpo8ek3GRLYPkkppzzjr735NoYc17\nZbxs9v4DritgGQqoHUXyk9LzoPoKifyV/0pa0f0y5djcmnWW3dkybrBOgGRKHrxRImnHkmX8bPXV\nsm7FFbux/Se9mioc99yIZ35Gb5yU4fL79PYfkmDvCVCokdJbJePxs0w9LhGkKetlru+GQ5MByDlz\nxgmlLZ6sCkHnHMOyn0Z+MgSAOnOkF8IFv3s58q4RXWkc9AAbbpQI1G+XzAKgwzUSLa3whfsjov6x\nMWzqOA2Ah/ZJNMx/Sel6cZiFMWYyvZccz6Nfk/+Hq0JP556/jIz9+cdMXvrcYABa1PBjXrTMc9+m\novRKq/6uZ/VWsbWXG6VpQ94tdJ3mn4wAIOYl2ffx78iXe/ONkwvdRpUdj30QnfiMdAF4vrkEbSsl\n2zjWWC5+Qc3lAvdGM5my5J2achO74HQEN4QVnPwkwybdLlZlSheaTiFZYN8u9jZJfBG3uMyrUWLW\nIkzNkfZKe4ZWfMv+myR5eGy/pMwv94tc3Dzlhq2sXT1MHtjn2pMlRCyVbstmq2/gUnmINLrAXMjf\nZ+VikJYlCVz6hEv3qFsj5Kb/1s+nFnuaA1eydWjJ5NflpuSLk9IIFD9KjvOS7Lf2XTeWVdHcKvNy\neYh+udrHDstP27uKlZ/u/ukDCmJM1XL4xrwb6NZvyc1Ijd0lvxkxbhxGffl/ucv+WyAPceFHU0r8\nuc6Q/JQ0EjS/wIPIqgS5edu6UvbnTZ8/CkD9V2QYgSc8gFyIf0VpqG27IJW5VaTLXoc1DzisE/yj\ndEUdecPg3Cl6/BtL0pfILTvkZ47jNc0s6fRyrpAb9TR5fmJBB7k5bRBgdEH25+cMef1MUm8Aju+S\n69LG3nLz+9xBuS6/UWMNLUJ3AjCu7UwAnn5kMAC1X3P/Dbw1xJL7uvmKwQDUfaXk5Wo4fBXfXyPX\nuL4RRwE43usUABW+KPHHlpktL1S8+EpFkNldumum18m7pa66VgIJtrWe2RU7s5MM+fp1/CSH5Usy\nInj+ZWl8CDzt2Ph5sp7cjwfZ8/s88fg0TuTINzliv+fdfdnat8D2stwnJdhjNcZD9pz/qgEwbXAv\noldJQ5HN3pDS6JH1AHSfez8AL70v07q0DrZyzUbpevxLs3JOL7+zNJh5Hztue99h2fZ35Bzlji66\n2jVXKaWUUkoppZRLeWxENHzWKvvPvGXl863zbo1OALzcIVre/207b3SKLfDzAjKkHSR8w34AKi+b\nzSVB9qRHaebq1nl8oCRy+mPQW1Twk0joykxpkV/3siQvCj3p+YlrSsK/aSMAXq32FQAfnzRvkpqM\n3m25s+Y3QF6X3IK65jZbLIlfqi6WJr3gE7LO052kHemfvhNz193ztCRN8YTW9fyOPX2a2gHSevro\ngzcAEHis+F2JA2pKdOyTugtz062bWepNBUfTbtnW2/7KM5Pc7J4gSUi2tf0UgFGHWhL1ibT+l6Zt\nfG8n6bXSITiHZiukW2ddD+vuBeDfJI5frh5v/00iYq8flej2muMyvcXMBgtz14+zJ+r6sP97su60\nGwHISd3piuIWmzE1WOYsiYg+U+VXGn0rYcH4OQXvZyMaCmBNLtmUPZ4kZXpLvmz/IcA53W9lX/dL\nvRaA1ZtjiH9IeiBVPSX1r2pf876EawA4NEJ6gDzynj+jqi8FYHmGRArXPSBR095fyPGQvXuPcypT\nBI2ezove+a8tm4jPs6vlPNa3s/T4GN5Upo74nkpl8vml8c5lM3Nf/zFdho3U4OLnmh1fyn3WhMvk\nPuSSIBlWUd0/b4jU9iy51t046xEAGjzuGT1bjB4nr733gcPyfjuuB+Dk6DpUWlLw1DUVYmMAaPmN\n9HJoHORH/DypX9ys4g+vcbZDbcJZHS9djo1eZyfsPY1Gfy29LqNXnl9XY6qtwEVrABjwk9yDbeo5\niZGRUvcPv5JrU0y/9c4qvtPkj4a6m/nv4pRSSimllFJKmYrHRkSLwkj/HT5bflqB8FlHL7jNwbsk\nmtg0KIC3/pXoWvQnMu7ILGNWjlwq/faNaCjAHUvvAiBurndGQg17r63s8Pva9Hr2VxmuL0wJGVHd\nl8dNpXXQWWOpwzpGAqJRS26m8RP2BC0nTzqs02ibTO+T2EuOg7bBZ/jx/jcA6Boi01xEvyoRR6OF\nzx2MqZi+ueRNPjvRHIDAX0qeVCnpRRmbmGWzckeaRByshw6XspTuc0MbxxbVEzlyLGe9INMc+Hlo\nRNRmk/FkRhR/1dFo/DMOFftz/MpJ5GXLK5LEZW4vmZIph0Dq9v2nLIrqFEfaViY6QBJN3bO7IwB7\n2kmOAr9wGUSVcN+DPH731wD0Lyf/Nx3tp+35s3cBkHSDZyUx8q8kkarNL8n5ZUvjKQCszYT4F+Va\nmf9c5C38wiUav+1FGT+XfNVk/Ozn5tX2hEz950kik0ZjJAoad3yNQ3Kqc11Sbi8APwdIJGnNmwlU\nHieRo97hx+1rWQra1KX8mscD0Kniz2zNkjHLVTZkXWiTIqv0m/2AL1lOSafwLy/968L9MlmUIfu8\nxjsFR0ItgUGc7SzXrWff+wSAjiFy/TKibImZEgkdtLkvj8bI1E297OeAKb0lEjx+Wh+gaPk/nOnY\ns3J9McZMXr/5JgD8H5f/E/+/C0/WdDxBrkmjq32du6zOosLWdj+/a47mJloyEi/dmdILgOjnCo76\nFiTufrmvfveKpjwaKfdj/ZvIuPgVBJVVcX2WRkSVUkoppZRSSrmUqSOixWFkeJz0jGQIC7T4880E\niaZU3l/0lhF3OvuzRP9Wxr9tXxJCi5XST73xY9Jv3fPylpWtk00cW2nXTZKMhhUxxz4EyLGnx8+L\nhuYZsvM6ANJvk3FIcXsSC92nRsvqsE9l/MKae8dT01+2+2uojF27+Vs5Pmzrk8um8CXg1/sIALUC\ngvl4utSvdhHG4eRnRJK/uFrGtmTastg1TqI24ZmeNz6lKDKvb8OkqA8dlu2xd83w++1vN5So5H6I\nn8vQpRL22JUuEf2zH9codP0DV0oT9fWXrQPgu1pT7O/ImP0O626nEp47ztAanDfFxYYPJIIWaT8P\n5ZySzKA1317B1z0lo2a/ct/LhjZpoT+YKZFg2xn39VYoyL4BMs51Sx8Zu/jdKYmQftzjWqyHd7it\nXK5wvJc9k2hfyUbvRxiLMyR0NHaYnEtjF8lYv4LOy5YAObf7NbJP6zM3EoA3P5MM0JcEHQIkiu5v\nkTjAJav+B0DUIff93267Q7LH3h5xmCs2DASg/A+r3VYeZ0t9uBkAV4QspsmSQQDE4ni+9bePh9wy\nvDpJtzpO/bE4Q8bHD/tpMADxE+QaF7x1B5ORa9K7i+We8/t4md3htboy1jooqUyrUiypM5qzqZVE\ndfdkS2TU71n5ftv+3lDodsZUL7EPS+H97DGsO3deTagH9sILiKoFwGONfjnvvZRvJJt3dYrfi2ra\nvGt49M7NpSucOo9GRJVSSimllFJKuZTPREQ3PxIFQJtgGY+x6WwGkUmn3VmkIguoHw3AS7GSYbWS\nfWzo2kyo95K0y1qPHXNL2Vwps3sb5nWVlskXj8h8mZGzpRWvsDE6ZvHMwdYAnLxLxsBa9xQ9EhQ9\nW1pjn+vdjrE1PKcV27+q5I4cFbcgd1ntV0ue/XTzMGm1bx0sx/zkY00In23OSKjhYJvzM3b3/P5h\nABri2XWr9q5E35dMlfNR59AzfFx3CQB+9nFvOeNsBW987jo4rvNVuoxDqvxMgEd/r8vdvD/39Ylu\nEgGN/OT89Z6v9539lWO77/K/ZVxe3DHPiiikX+Y43n5C6tUAhG717mgogM0+VP+MLW/cZnqOHOcH\nLpOxYBk3tQUgtuE5+/+MfAf61pPxdcMrfg7AmrOyTYdg40gOy93mjzOyLOpl+VvuHMf/SHc5R2/N\nOkPQZCMPg/fub0vzvDHOgTtCC1zHmGN0c+fJueeh/indATj5hNxPNlwp5+iCouPbU+y9QeJLX96y\nMqhJYu6YyZ3Z9nko/rx4JHTLeMm0O6+uzKFr/H/sfLMRYR54nTp2hWQtvyViXu6ye3Z3AiDKnvG3\ntDlhmoVKduvE+l3k81LSSvmJvsvrH0Qzb5BuUX/d8o59iXyx7n/oIUJXeNYNQGEafC0JD1oFOd7I\n9Ft8H3HrPefBw9n2dAmgeZBc8O9Iky5U1U6Zt5uEkegAYMOlxs14CboiWuRGJsAvx+EzAfaNkZ81\neuffyPksYbKvuoXJ1DptVw+iBiXvIlwl+l+H379MbU0V3Jv4obSCWuU1ICWflYax+InSsODp3ewD\nfpWEHROukAvxS5dHs6erHMfbe0p6+MRMOTYHLLrvvO0bfiY33gu+meaw/I2kbgBErffMieAN6bNr\nQlN5PbiJ3IwtayMPKYdbSdc9W49/aRYo15nkLBlW0NQ+jcuc7tKo9mS7u+VDLnBD6EpfdZhqfyXX\nm1lNvgCg/bjHiPlOhhP4Ly08oYmZVZonx9w9g/oD8EX8F/QKl2P65vul67jV5tg8kmnLJtiS/1ZK\nfs97ABXZWOm0QaaNiBwu33Bbiucc5x8c7UjI9+a4LyqN+GoHC33PkiBf6jlXvGdfEkjTpfcA0HCo\nXL8sZ4o+Zcfzh+QeNGSpJF7z5MY1gzEMJvlB6U68uedkh/eX2Lsml1uR6pHXqcOXnp8AbMdYGXIQ\neqBsju8e4ZIYdVxraXCIMPmDaOwj7pteSLvmKqWUUkoppZRyKa+PiO7qLs/aERaJhBqTUIctXE/h\nncY8w7E7ZNqLMdWN5ERSB2PKisZPbPfI1ihnqdrsUG5rdMA890+GXVJb7pfuWca0F6WVdpN0pZpV\nNZEse98y47NrjZZ13NEKm/OvTE/w0mGZKPx/DdawrKYk8SjOVBVGorE/Ws6wL5HvdMafVcCkEdEz\nPSRytqbNexhT92zJqgaA1WRdII1ptMK+PUic5OXg+vsudVgnjvNboY0pI4wuui8fkQQi9R6SCLqn\nT6dV47tUtj4tEcKRlSWJx5NzJWJybnfj23bcAEDGCOmq3uerpQDcWX43ADtGyPHcwDPmu6dtsHQX\nN84hxlCQzbdNJutWWdZssUS4K6yW9/6rLfUtnwJVNpxy+LwjzWV6jOpLZfoaTz6+c9LTAQjuKj/v\nqX4TyS9EA9A1QSJaW0/I93Tn3ioA+AdZ6dVIotlv1Fhzwc9vsuQeGj0mPZyyDxZ/qqOy5l9RIl7l\n/Pa4uSSuVTtMrk1++IHF8U5w6wi5z2ocKN+DhNUDaNBfEhkV5zoaGCHnhlPZ8nk5Z86UpshlYnZq\nS0ZWluO4VbB8T6/ccH652obJibxzqLyXv96Prb8FgNoHPSeafy5rmJTY75xYW1klVTJ6nWV5+gOE\niWhEVCmllFJKKaWUS3ltRNSYJH3glb8DcDJHWnYOvVofgOBMzx5bGRBViytHyLijCL9gh/dWJsUC\nEHfMs+tQVgJiZNqatxp9w4cnJDoWOc0807XkN+rK+aXaPqBObQDSEyRF+ft3TjlvncRMiVRYzrov\nrmREFxbtlcjX8pbT2f+9tMAv/6B9odsdbyJNjRHREhlrVytNPi9fu6zFxC2SGVWkVfXcMb1PrJWJ\nxWPwjLGCzrZrtNTdiB4ueqUjABG7PSQ0eBHZ+w9wz0hJLPXJW+MAiAuU6J8xRUvsoruJf0DGseec\nkqjp2F97AjC0t4xBe721RB8+anEDOW6cZskQM1/GrG7t8f557xnH65Zr7FMOXVP0z018SiLfDyfZ\nx0j28PzeDNaDh4i7XyKXafZlQewEoKH9J8CiOU2A8yOiadky7rv3u0/INuMTsWZ7Tqx/z1AZD9m/\nnCQZ++tUdJn/jczrTzj8fjonqMz/RnHl2CQGk0MO2BzHE9asfjzvPaBJ1YMUJxWkMe3Lpo4y9r3j\nhlsBKO8ByZ9qDNhLr7l9APg+XhL5GBHSglz55IMA5PST8ZDLW04HoNqHYYVu4wmaN08Dzr9nKAtG\nTxFnfLav0oioUkoppZRSSimX8tqI6LYXpKXv+yoSLbpx280ABJtkkubkZ+owt4Zj5KzzP30BGRsK\nnp9Vs6xsu1cif+2C4e6/OgNQh43uLJJbJY2RLG2buk46773Z/8m4pfcel2MlJNn9GRArjZHo7FUv\n9GNOs08BeH104RHtNZkSdbHa28laB521v+PYcl333X9M2yaZ2ft47msjW27tj86fysUbHblHouEb\n2kkmxjT7xOqhh88Wuo2nivhGeq3cyaMA/Hur7MszJ+zjzEbuwHrKccxko6ckMnp1Q4mA/9x0NgCj\nR/sRdZPzy3wxjYbLeLhu30im0EGT5DoU5pdJjzCZBD5/du6iaBsske/fW30JQNM3R9BgpHl7thhS\nX23PX22MrPyO0b5b3pBIaK3JMm2ViTtxlEh2lwRmtDKuU/KdmPO6TAdUAc/s+VBxqPSeW7VczseT\n6s6n/euPAxA3UaLg2Xv3Fbp945myzkGrnNdCJkTa33F/RDQnPR2ulp5KXfoMA+BQgmM8qlKyjQpf\nyr45/LlkNt9sz8/w8YloAMI2ydRFnhPbd72d2XK9MuN1y9NoRFQppZRSSimllEt5XUT0xIB2AGy4\nbSIAO7Jl7rb/XpdxdcHsL3hDD7O21zsYLYiGCsMk/pN9rDgjFswvp05eVreM4yFuLIl7BS6tCcBr\nNWcXus6ney8HIGS++yOhuRJlDEqF62FgpxEAHG8YXOjqlT90jJLs/VZ6N6y97FOH5cYYVDPxj5Os\nwWvafGEs4cf/JFts4C9r3VQq1zp97X8Ov9+y7i4Aqi0x79yURmQ04hvH5QX1WjGO25NzZL8bc5G+\n3nw2U2p2AoqXVbqs2exjGI3j8av4WrnvTbxFxndaA6V3wuWPy3lmbI2i9zQyMlnWbmGOa3Fh9o2U\nc+1P/d8g1OI4Zm7CMcnjUOOTdYA55o4sS9ldEgD496FTxAfKuX7Y3g4AVJwp33N3RIeN8ZsdK/xa\n6DpGtPP1a2Ty7RazU9g4QO4nh10lPbL23yBRTutRmdv6+EDp5XHFw6t4vvofACTMkChqg4WeGfkN\nmyPnrOg5ha+zuctHQN54yMlbrgKg1u4k5xbOQ93Ve1Hu6xs/GQlA3SUr3FWcEhu0syOf1VvmsGz7\nO/Ls5I7TnEnsAAAL6UlEQVT5RDUiqpRSSimllFLKpbwqIhoQVYuHn5sJQLBFqnb7+oEAVP3RHGND\nLySrumQcDTwbVeg61sNHALBlSt9+S7C0RvpXrZK3TtWKAGx7rODsdTarhfgH7eNQT54sZalLb8pl\nX+S+jvqx+GOTPI2/RVoXzx1ndfJ/7RzWGfPix0DePF7nrp83/+j5/xe2LnvLsqhlzn+ptIZXXlr0\nbTLSJAM2lzkut3VoieWPdWVTMBc52FnmIDx3309aInMbN2SVW8rkah8kfA7AfquMp6w83rMzMDpL\n1Q8kmnhZ9/8BsCphOg89Hg1Ag8fcFxG9kPBZjsfo/BYSCRo7UK6vp21nSVh2PwD1PpJj/MgI2c95\nvQDMLatrawDmPvAGAHUD8o7fXfYsud89KeMgg0979n1H+TS5lhjZfUvLEiD3Xccfkaj/mktn8HNG\nKABbn5PQf1DWhedadSbr9lQAZhyQeZz7NFhIvSt2AeBfvrysY7/nyU5JA2BtKz86DpSePJEbZGy/\npYr0tEudJFn8N3WUcbAHrRl5kdDHPTMSWhT+TRvZX0mvCGM8ZPWJ5uiRdup56cWx5hN/WgfLMb7r\nm0sAqNu38CzBF9MmNJXETOkNEv3mesD3ejs4g1c8iBonvxbf76FvhKSZ/jJdbviqP2ek6Ta/BbOm\nXXSdy//uB8CRg3JSrVRVLgirEqYX6281GfUAAPWfcF8yiTM95WJxRYjRzdQrDlfGzpTJoG8dOj53\n2bI3JXFL3kMm9t/P3z7/OoZmi++jIebt3lgoe44iv3wdOMz2EApwJtIx4dLazLM0fl0mk/eFxA97\nnr6cDsFyjP6ZKTfw/ibuklsqOfI9rvy2/D8c+TyD5NvlPNBz+iAAbGs9c8J4Q92fpMETae8lzBJE\n8lXSiDawnjSw/BD9k31tx+/vrgORNMydFMU80nrIA3b0OQ+gRqPKoIcfAyBsgTkalcJnSzkXvtQY\ngAYhh9lWW7qMZ++5eKNmzhUtAUiVvDfc3FjOya9Wm5G7zquP3wFA6E+eM1zkzF1yfzRudnzuNCYP\nLZauw4nvS+NKxL68M/LhNnIH2WZECgBv15JpAY1r0lR7Ep9P3+pBAxNPLWdIGe0YpOj7twyfqGGS\nc7Xfb5Jwbfj4B1j95LsA/HyZTJc1uLM0KhTnupM6ozkAHULW5t5jR57y/Omn8jvdR1rzP6v3gZtL\n4ki75iqllFJKKaWUcinvCDG1kG4EL1X7PHfR5Fdl+oqK683ZOnVjUn8WN5tV7O1WtPqq0PdO26R7\nRZbNMT58/YbBAJxYl9d9N+p398dndvWScKDRzfrFI5cQMU+6ipg5DX79mdJ9OnFACG2Dz1xk7fMl\nZkr3mKkHJHHAsWEynUt86nbvnNLHvrO9YQLpavm6Tn93slVud3pf0L/fYnLsO3TomsEA1EO6SvlX\ntk9zUK0yANbkbS4vnzsYrfed/m8kSUMkIpr+ikz9UL6vdEv31MRcgWtkH7X7S6IEf16ad/35PPpn\n+ytp7860SXfGHkmS8Ch+xA5Tna+M4/Pvm4yeLHkJ1zr9Lj2IGswxRyS0MMMqpnLwe4kWrvm37kXX\nHxszFYCWQY63kmvPyp4dmDiUBr9uBjxrujnrVplKZdmNTam0QKZXeqfWcnnzxeUO6/rhV+i1p9nv\ndwIQ+6icwyP3mvN+81y29i347rIp9t/kXsOyuJL7ClQKNZf+S+suA4C8oQF7Okmd6i25+PanbpYI\n4teXSbKqlZnBRL5sju7JBYl5ItndRSiQRkSVUkoppZRSSrmUqSOi/k3iALhnxrzcZU2mDQcg+nPz\nDhQHCO2WStNXpZXVVsheKhf/b6FjP5sul5Y6267w3GX1Z9mnTUh0HKxdiW0OP93NSBrwZIcfHJZP\n/7Ej9bPN3+JoTZKxBc8/ehe7e0pL69buRe+zP2zafQDUecVIG+7d0/nkhDi2Rh+2ZrqpJCVnJA27\nsdZ6h+VHz0bkJhbzNTlWaQc99IBMg3HDXRKJmJsi0xRF3eSecrlL7NTdfN5Xejcsu0R6w1zXYggA\nfr975nhoI1Jb40GJmPSc1otnohcA0N6eJGT2f9LT5tkfbgPypgfwpAjZhfhXkro9vEqOzwiL49RT\nrx9tTMO75dpp1j4bn77VA4BDDy1jTFX7Oarq+gtsYZCbk2z73lwvna4YMFPG4cU8tdKj93N2Shpz\nO8mY2Il3ynQtp2Ikcv/TdRL57vbTw+d1wWr0kfRkil69QT7HFYV1kUNtwokJkKifEQkOOGPOPmg5\nGzYT9az0mJwzR3o0fDf4TQCuq/IoAA2H5/VisCRIQq2D7SUx6AePTQCgcZBcq+Ln30Pcn54z1rmo\nLjQ29Mrh9wIQO8d9z0waEVVKKaWUUkop5VKmjohuHmZvhQ3Lm2Kk9lJ7k5zNnC0454p55uLRvx4k\nFLwtG8q6OC6TY48QJZ2WFNzX7JV0+Q1f3eTRravFFTovkTh7ML9jP4nkBw4+CMDCpjINUdeNt5Pz\nqWSAttkTrkavOwyYJ6JQWl9c9z4AyWeldbbfp08AUBcTTSRtlb01NfkKAB6+PA2ApbtjicKzM6M6\nS3LHTwDI6Sjn6qbLJPoX+4KM2fKV49uQvXsPX/eRcd8Df5Hv/5GREnmp9rvbilUk2WkyBQZdYMQI\nSaGa3kbGucaPkvFzsTvN2UvpSK94ALqGyaAya75bix/GdCL8lLnHhkbaM72uXhbHuLlyzD1a6eI9\npOJ/k+9s0D+SQbj2a3JOjsE8PZesBw8BEDX2kMPyB5EsunGcPwWP+e8uC3emii03Ejr+3yYAVP7Q\nPPszP+umLQD833WdAfhgqtRtYY9xAHx9ZQIzpncB4KN7JMNuq2DHvg3XJclMB/HvpZu218O5Gsy8\nL7dnSpgHTBunEVGllFJKKaWUUi5lyoioMb/k4p5v25f45oTo3soYM7dFAqEEsRPw7ghJ+a/s0QJ7\n0sk+yDEeTgqQ4rCuN/8/FOTF1F4AnJoSBUDd2SaKhNrZsmUUUfRTEu1r/JpMvGhZV85tZXKHn569\niqSnZQzoylUSaYqfsA+ABgek5dp6pviZpL2FkSn4tpSuAMxv9REAQ9vZJ2r80/N7ulSfKN/P6vbf\nzT5+7ubHfwHAmi/bfOx8GasfN9v9EYWyYt2eyi/N5Jz0C5dedP36eObYZVVyA3rnpZOdNu8aAKJN\nFOEuTHZKGgDB/aoCcF+rhwAIfPIAax+UsaDx84c7bBPzrXzng5fIeTcn66wrilrmwuzZvLvNkXl/\nY/Gs3ikaEVVKKaWUUkop5VKmjIju6+APQN0Ax0jol+nVCDwpLRbe3IdfKZ9y9R4Awtnj5oKUnnV7\nKgB1+7q5IG4SMj+Rw/PltdEqa/aImTOc7iNXsFUrZJz8sUaS/bySZzVk+4QWoTL+1d8i7fZ/npE+\nKU3ekDGFevwqbzI7tSUjK/9z8RVNynpYcmwELpKfLIJetAEgjoIz4urzhHOZ8kE0v9eOyoDqld2i\nse333i+QUkop72c9chSAqXH1AajkBV3jzOrhL4cCsPnuKQAMmfYgAHVSzDdEQKmLsS2O5JnaMt1H\n9TW+NhBIuYN2zVVKKaWUUkop5VKmjIjWf0pah69/Kv9g+gOuL4xSSimlvFK90RL57DZaEn3UMdO0\nUUoVU/WJK9g4UV6HFtJVVamypBFRpZRSSimllFIuZbHZdBiuUkoppZRSSinX0YioUkoppZRSSimX\n0gdRpZRSSimllFIupQ+iSimllFJKKaVcSh9ElVJKKaWUUkq5lD6IKqWUUkoppZRyKX0QVUoppZRS\nSinlUvogqpRSSimllFLKpfRBVCmllFJKKaWUS+mDqFJKKaWUUkopl9IHUaWUUkoppZRSLqUPokop\npZRSSimlXEofRJVSSimllFJKuZQ+iCqllFJKKaWUcil9EFVKKaWUUkop5VL6IKqUUkoppZRSyqX0\nQVQppZRSSimllEvpg6hSSimllFJKKZfSB1GllFJKKaWUUi6lD6JKKaWUUkoppVxKH0SVUkoppZRS\nSrmUPogqpZRSSimllHIpfRBVSimllFJKKeVS+iCqlFJKKaWUUsql/h9ai55FX1WawwAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdb3cbcf2b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "slice = 15\n",
    "predicted = model.predict(X_test[:slice]).argmax(-1)\n",
    "\n",
    "plt.figure(figsize=(16,8))\n",
    "for i in range(slice):\n",
    "    plt.subplot(1, slice, i+1)\n",
    "    plt.imshow(X_test_orig[i], interpolation='nearest')\n",
    "    plt.text(0, 0, predicted[i], color='black', \n",
    "             bbox=dict(facecolor='white', alpha=1))\n",
    "    plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more Dense Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv),\n",
    "                 padding='valid', input_shape=shape_ord))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.1922 - acc: 0.9503 - val_loss: 0.0864 - val_acc: 0.9721\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.0902 - acc: 0.9705 - val_loss: 0.0898 - val_acc: 0.9676\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdacc048cf8>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          epochs=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.0898462146357\n",
      "Test accuracy: 0.9676\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv),\n",
    "                        padding='valid',\n",
    "                        input_shape=shape_ord))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.2394 - acc: 0.9330 - val_loss: 0.1882 - val_acc: 0.9355\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.1038 - acc: 0.9654 - val_loss: 0.0900 - val_acc: 0.9679\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdacc064be0>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          epochs=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.0900323278204\n",
      "Test accuracy: 0.9679\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more Convolution Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv),\n",
    "                 padding='valid', input_shape=shape_ord))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Conv2D(nb_filters, (nb_conv, nb_conv)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "model.add(Dropout(0.25))\n",
    "    \n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.3680 - acc: 0.8722 - val_loss: 0.1699 - val_acc: 0.9457\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.1380 - acc: 0.9508 - val_loss: 0.0600 - val_acc: 0.9793\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdb308ea978>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          epochs=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.0600312609494\n",
      "Test accuracy: 0.9793\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "The above code has been written as a function. \n",
    "\n",
    "Change some of the **hyperparameters** and see what happens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Function for constructing the convolution neural network\n",
    "# Feel free to add parameters, if you want\n",
    "\n",
    "def build_model():\n",
    "    \"\"\"\"\"\"\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(nb_filters, (nb_conv, nb_conv), \n",
    "                     padding='valid',\n",
    "                     input_shape=shape_ord))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(nb_filters, (nb_conv, nb_conv)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "    model.add(Dropout(0.25))\n",
    "    \n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "    model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "              epochs=nb_epoch,verbose=1,\n",
    "              validation_data=(X_test, Y_test))\n",
    "          \n",
    "\n",
    "    #Evaluating the model on the test data    \n",
    "    score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "    print('Test score:', score)\n",
    "    print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.3752 - acc: 0.8672 - val_loss: 0.1512 - val_acc: 0.9505\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 2s - loss: 0.1384 - acc: 0.9528 - val_loss: 0.0672 - val_acc: 0.9775\n",
      "Test score: 0.0671689324878\n",
      "Test accuracy: 0.9775\n",
      "5.98 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "#Timing how long it takes to build the model and test it.\n",
    "%timeit -n1 -r1 build_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding Convolutional Layers Structure\n",
    "\n",
    "In this exercise we want to build a (_quite shallow_) network which contains two \n",
    "[Convolution, Convolution, MaxPooling] stages, and two Dense layers.\n",
    "\n",
    "To test a different optimizer, we will use [AdaDelta](http://keras.io/optimizers/), which is a bit more complex than the simple Vanilla SGD with momentum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.optimizers import Adadelta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_shape = shape_ord\n",
    "nb_classes = 10\n",
    "\n",
    "## [conv@32x3x3+relu]x2 --> MaxPool@2x2 --> DropOut@0.25 -->\n",
    "## [conv@64x3x3+relu]x2 --> MaxPool@2x2 --> DropOut@0.25 -->\n",
    "## Flatten--> FC@512+relu --> DropOut@0.5 --> FC@nb_classes+SoftMax\n",
    "## NOTE: each couple of Conv filters must have `border_mode=\"same\"` and `\"valid\"`, respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/sol31.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding layer shapes\n",
    "\n",
    "An important feature of Keras layers is that each of them has an `input_shape` attribute, which you can use to visualize the shape of the input tensor, and an `output_shape` attribute, for inspecting the shape of the output tensor.\n",
    "\n",
    "As we can see, the input shape of the first convolutional layer corresponds to the `input_shape` attribute (which must be specified by the user). \n",
    "\n",
    "In this case, it is a `28x28` image with three color channels. \n",
    "\n",
    "Since this convolutional layer has the `padding` set to `same`, its output width and height will remain the same, and the number of output channel will be equal to the number of filters learned by the layer, 16. \n",
    "\n",
    "The following convolutional layer, instead, have the default `padding`, and therefore reduce width and height by $(k-1)$, where $k$ is the size of the kernel. \n",
    "\n",
    "`MaxPooling` layers, instead, reduce width and height of the input tensor, but keep the same number of channels. \n",
    "\n",
    "`Activation` layers, of course, don't change the shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 0 \t conv2d_12 \t\t (None, 28, 28, 1) \t (None, 28, 28, 32)\n",
      "Layer 1 \t activation_21 \t\t (None, 28, 28, 32) \t (None, 28, 28, 32)\n",
      "Layer 2 \t conv2d_13 \t\t (None, 28, 28, 32) \t (None, 26, 26, 32)\n",
      "Layer 3 \t activation_22 \t\t (None, 26, 26, 32) \t (None, 26, 26, 32)\n",
      "Layer 4 \t max_pooling2d_5 \t\t (None, 26, 26, 32) \t (None, 13, 13, 32)\n",
      "Layer 5 \t dropout_6 \t\t (None, 13, 13, 32) \t (None, 13, 13, 32)\n",
      "Layer 6 \t conv2d_14 \t\t (None, 13, 13, 32) \t (None, 13, 13, 64)\n",
      "Layer 7 \t activation_23 \t\t (None, 13, 13, 64) \t (None, 13, 13, 64)\n",
      "Layer 8 \t conv2d_15 \t\t (None, 13, 13, 64) \t (None, 11, 11, 64)\n",
      "Layer 9 \t activation_24 \t\t (None, 11, 11, 64) \t (None, 11, 11, 64)\n",
      "Layer 10 \t max_pooling2d_6 \t\t (None, 11, 11, 64) \t (None, 5, 5, 64)\n",
      "Layer 11 \t dropout_7 \t\t (None, 5, 5, 64) \t (None, 5, 5, 64)\n",
      "Layer 12 \t flatten_6 \t\t (None, 5, 5, 64) \t (None, 1600)\n",
      "Layer 13 \t dense_10 \t\t (None, 1600) \t (None, 512)\n",
      "Layer 14 \t activation_25 \t\t (None, 512) \t (None, 512)\n",
      "Layer 15 \t dropout_8 \t\t (None, 512) \t (None, 512)\n",
      "Layer 16 \t dense_11 \t\t (None, 512) \t (None, 10)\n",
      "Layer 17 \t activation_26 \t\t (None, 10) \t (None, 10)\n"
     ]
    }
   ],
   "source": [
    "for i, layer in enumerate(model.layers):\n",
    "    print (\"Layer\", i, \"\\t\", layer.name, \"\\t\\t\", layer.input_shape, \"\\t\", layer.output_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding weights shape\n",
    "\n",
    "In the same way, we can visualize the shape of the weights learned by each layer. \n",
    "\n",
    "In particular, Keras lets you inspect weights by using the `get_weights` method of a layer object. \n",
    "\n",
    "This will return a list with two elements, the first one being the **weight tensor** and the second one being the **bias vector**.\n",
    "\n",
    "In particular:\n",
    "\n",
    "- **MaxPooling layer** don't have any weight tensor, since they don't have learnable parameters. \n",
    "\n",
    "\n",
    "- **Convolutional layers**, instead, learn a $(n_o, n_i, k, k)$ weight tensor, where $k$ is the size of the kernel, $n_i$ is the number of channels of the input tensor, and $n_o$ is the number of filters to be learned. \n",
    "\n",
    "For each of the $n_o$ filters, a bias is also learned. \n",
    "\n",
    "\n",
    "- **Dense layers** learn a $(n_i, n_o)$ weight tensor, where $n_o$ is the output size and $n_i$ is the input size of the layer. Each of the $n_o$ neurons also has a bias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 0 \t conv2d_12 \t\t (3, 3, 1, 32) \t (32,)\n",
      "Layer 2 \t conv2d_13 \t\t (3, 3, 32, 32) \t (32,)\n",
      "Layer 6 \t conv2d_14 \t\t (3, 3, 32, 64) \t (64,)\n",
      "Layer 8 \t conv2d_15 \t\t (3, 3, 64, 64) \t (64,)\n",
      "Layer 13 \t dense_10 \t\t (1600, 512) \t (512,)\n",
      "Layer 16 \t dense_11 \t\t (512, 10) \t (10,)\n"
     ]
    }
   ],
   "source": [
    "for i, layer in enumerate(model.layers):\n",
    "    if len(layer.get_weights()) > 0:\n",
    "        W, b = layer.get_weights()\n",
    "        print(\"Layer\", i, \"\\t\", layer.name, \"\\t\\t\", W.shape, \"\\t\", b.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Batch Normalisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_A little while ago, you might have read about batch normalization being the next coolest thing since ReLu’s. Things have since moved on, but it’s worth mentioning because it has been adopted in most networks today._\n",
    "\n",
    "> Central to BatchNorm is the _Covariate Shift_. Formally, covariate shift is defined as a change in the distribution of a function’s domain. Informally, it’s when your inputs change on you, and your algorithm can’t deal with it.\n",
    ">\n",
    "> IDEA (in short): Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.\n",
    "\n",
    "\n",
    "\n",
    "**Suggested Readings**:\n",
    "\n",
    "* [Paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167)\n",
    "* [Alex Smola's Article on Covariate Shift](http://blog.smola.org/post/4110255196/real-simple-covariate-shift-correction)\n",
    "* [Understanding the backward pass through Batch Normalization Layer](http://kratzert.github.io/2016/02/12/understanding-the-gradient-flow-through-the-batch-normalization-layer.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to BatchNorm in Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "\n",
    "BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True, \n",
    "                   beta_initializer='zeros', gamma_initializer='ones', moving_mean_initializer='zeros',\n",
    "                   moving_variance_initializer='ones', beta_regularizer=None, gamma_regularizer=None,\n",
    "                   beta_constraint=None, gamma_constraint=None)\n",
    "```\n",
    "\n",
    "#### Arguments\n",
    "\n",
    "<ul>\n",
    "<li><strong>axis</strong>: Integer, the axis that should be normalized\n",
    "    (typically the features axis).\n",
    "    For instance, after a <code>Conv2D</code> layer with\n",
    "    <code>data_format=\"channels_first\"</code>,\n",
    "    set <code>axis=1</code> in <code>BatchNormalization</code>.</li>\n",
    "<li><strong>momentum</strong>: Momentum for the moving average.</li>\n",
    "<li><strong>epsilon</strong>: Small float added to variance to avoid dividing by zero.</li>\n",
    "<li><strong>center</strong>: If True, add offset of <code>beta</code> to normalized tensor.\n",
    "    If False, <code>beta</code> is ignored.</li>\n",
    "<li><strong>scale</strong>: If True, multiply by <code>gamma</code>.\n",
    "    If False, <code>gamma</code> is not used.\n",
    "    When the next layer is linear (also e.g. <code>nn.relu</code>),\n",
    "    this can be disabled since the scaling\n",
    "    will be done by the next layer.</li>\n",
    "<li><strong>beta_initializer</strong>: Initializer for the beta weight.</li>\n",
    "<li><strong>gamma_initializer</strong>: Initializer for the gamma weight.</li>\n",
    "<li><strong>moving_mean_initializer</strong>: Initializer for the moving mean.</li>\n",
    "<li><strong>moving_variance_initializer</strong>: Initializer for the moving variance.</li>\n",
    "<li><strong>beta_regularizer</strong>: Optional regularizer for the beta weight.</li>\n",
    "<li><strong>gamma_regularizer</strong>: Optional regularizer for the gamma weight.</li>\n",
    "<li><strong>beta_constraint</strong>: Optional constraint for the beta weight.</li>\n",
    "<li><strong>gamma_constraint</strong>: Optional constraint for the gamma weight.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Try to add a new BatchNormalization layer to the Model \n",
    "# (after the Dropout layer) - before or after the ReLU Activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Self-Normalising Neural Networks\n",
    "\n",
    "_(excerpt from [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515) paper's abstract)_\n",
    "\n",
    "> [...] We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are \"scaled exponential linear units\" (SELUs), which induce self-normalizing properties."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scaled Exponential Linear Unit \n",
    "\n",
    "```python \n",
    "def selu(x):\n",
    "    with ops.name_scope('elu') as scope:\n",
    "        alpha = 1.6732632423543772848170429916717\n",
    "        scale = 1.0507009873554804934193349852946\n",
    "        return scale*tf.where(x>=0.0, x, alpha*tf.nn.elu(x))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Keras supports `selu` activation since version `2.0.6`.\n",
    "\n",
    "- To implement _Self-Normalising Networks_ the Keras way, it is no different from:\n",
    "\n",
    "```python\n",
    "\n",
    "x = Dense(..., activation='selu')\n",
    "...\n",
    "x = Conv2D(..., activation='selu')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Further details and Suggested Readings:\n",
    "\n",
    "- [Keras SeLU activation](https://keras.io/activations/#selu) \n",
    "- [Self-Normalising Network Original GitHub repo](https://github.com/bioinf-jku/SNNs)\n",
    "- [Keras SeLU Tutorial](https://github.com/bigsnarfdude/SELU_Keras_Tutorial) for further experimentation on SELU using Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Addendum:\n",
    "\n",
    "* [CNN on CIFAR10](3.1 CNN on CIFAR10.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
